[2024-04-22 16:46:20.594722] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=114][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:20.594753] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=26][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:20.594805] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=48][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=4, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:20.594877] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=57] will sleep(sleep_us=4000, remain_us=1972310, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.596481] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=21][errcode=-4725] [2024-04-22 16:46:20.596524] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=41][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:20.596566] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=39][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:20.596583] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:20.596605] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:20.596616] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:20.596662] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=44][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:20.596720] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.596793] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=63][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:20.596816] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=20][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:20.596884] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=64][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=134, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:20.596934] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=36] will sleep(sleep_us=100000, remain_us=21084996, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=134, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:20.599020] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=21][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:4, local_retry_times:4, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:20.599101] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=60][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:20.599136] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:20.599164] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:20.599199] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=21][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:20.602154] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] [2024-04-22 16:46:20.602205] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=48][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:20.602238] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:20.602257] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:20.602316] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=55][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:20.602339] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=21][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:20.602355] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:20.602413] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.602479] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=53][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:20.602503] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=21][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:20.602545] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=38][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=5, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:20.602613] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=53] will sleep(sleep_us=5000, remain_us=1964574, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.607829] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=90][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:5, local_retry_times:5, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:20.607903] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=56][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:20.607929] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=23][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:20.607964] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=33][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:20.608013] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:20.610785] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=48][errcode=-4725] [2024-04-22 16:46:20.610824] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=36][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:20.610852] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:20.610873] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:20.610889] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:20.610900] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:20.610910] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:20.610953] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.611005] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=45][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:20.611027] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:20.611072] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=42][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=6, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:20.611117] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31] will sleep(sleep_us=6000, remain_us=1956069, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.617258] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:6, local_retry_times:6, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:20.617334] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=58][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:20.617359] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=22][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:20.617378] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:20.617436] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:20.620592] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.620725] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=7000, remain_us=1946461, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.621816] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=87] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:46:20.631813] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.631945] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=8000, remain_us=1935241, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.642904] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:189) [14116][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-04-22 16:46:20.642982] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:445) [14116][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=49] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:46:20.643013] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [14116][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=30] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-04-22 16:46:20.643876] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.643950] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=9000, remain_us=1923236, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.656972] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.657090] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=10000, remain_us=1910097, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.671089] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:20.671188] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.671272] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=11000, remain_us=1895914, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.686072] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.686180] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=12000, remain_us=1881007, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.697089] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=29][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:20.701678] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.701780] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=20980150, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=135, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:20.702242] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.702333] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=13000, remain_us=1864853, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.719288] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.719377] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=14000, remain_us=1847810, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.721981] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=67][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:20.722075] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=88][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775580545744419, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775580721962) [2024-04-22 16:46:20.722154] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=75][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775580522001, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:20.737246] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.737296] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=15000, remain_us=1829890, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.739340] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC17-0-0] [lt=415][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:20.739413] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC17-0-0] [lt=69][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:20.739467] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC17-0-0] [lt=51][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:20.739524] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC17-0-0] [lt=53][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:20.742059] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=67] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=1) [2024-04-22 16:46:20.746422] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [14061][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=9] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-04-22 16:46:20.746511] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [14061][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=68] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=380) [2024-04-22 16:46:20.746895] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [13617][ServerTracerTim][T0][YB42AC175883-000616AB07F733A0-0-0] [lt=13][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-04-22 16:46:20.746977] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [13617][ServerTracerTim][T0][YB42AC175883-000616AB07F733A0-0-0] [lt=79][errcode=-4002] refresh sever list failed(ret=-4002) [2024-04-22 16:46:20.747002] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [13617][ServerTracerTim][T0][YB42AC175883-000616AB07F733A0-0-0] [lt=23][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-04-22 16:46:20.756093] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.756177] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=16000, remain_us=1811009, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.764306] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:306) [14058][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=19] ====== traversal_flush timer task ====== [2024-04-22 16:46:20.764408] INFO [MDS] flush (mds_table_mgr.cpp:172) [14058][T1_Flush][T1][YB42AC175883-000616AB0877339F-0-0] [lt=25] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f4d7e454510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713775597210958129, latest_log_ts:{val:1713775580545744419, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x7f4d953f4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2024-04-22 16:46:20.764512] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:338) [14058][T1_Flush][T1][YB42AC175883-000616AB0877339F-0-0] [lt=98] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-04-22 16:46:20.767499] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:20.776050] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.776147] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=17000, remain_us=1791039, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.784221] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=70] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=16647, clean_start_pos=880803, clean_num=125829) [2024-04-22 16:46:20.797133] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.797218] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=18000, remain_us=1769968, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.805468] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.805555] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=20876375, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=136, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:20.819440] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.819542] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=19000, remain_us=1747645, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.842305] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.842379] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=20000, remain_us=1724808, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.867007] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.867101] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=21000, remain_us=1700086, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.892526] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.892680] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=22000, remain_us=1674507, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.908260] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.908357] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=20773573, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=137, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:20.917158] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.917237] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=23000, remain_us=1649948, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.922176] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=60][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:20.922264] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=82][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775580750466388, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775580922161) [2024-04-22 16:46:20.922309] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775580722215, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:20.937876] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [14062][T1_LSArchiveMgr][T1][YB42AC175883-000616AB07773224-0-0] [lt=46] gc stale ls task succ [2024-04-22 16:46:20.939381] INFO do_work (ob_rl_mgr.cpp:704) [13683][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=52] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:46:20.940503] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [14156][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=28] current monitor number(seq_=-1) [2024-04-22 16:46:20.940610] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [14055][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=71] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=13061, mem_used=68631552) [2024-04-22 16:46:20.941442] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [14072][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=42] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:46:20.942187] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:46:20.944562] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.944652] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=24000, remain_us=1622535, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.952779] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF77C-0-0] [lt=22] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x7f4d7e495470, extra_info="") [2024-04-22 16:46:20.952830] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF77C-0-0] [lt=39] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.131:2882", role="FOLLOWER", total_group_log_cnt=7, avg_log_batch_cnt=0, total_group_log_size=860, avg_group_log_size=122) [2024-04-22 16:46:20.952844] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF77C-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.131:2882", committed size=860) [2024-04-22 16:46:20.953256] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [14014][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=16] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-04-22 16:46:20.953724] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=10][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:20.953806] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=57] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:46:20.953841] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=30][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:46:20.953873] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=29][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:46:20.953898] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=23][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:46:20.953918] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:20.953932] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=13][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:20.953950] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:46:20.953964] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:20.954145] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=173][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:20.954172] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=26][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:20.954185] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=11][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:20.954196] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=9][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:20.954206] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:20.954217] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:20.954266] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=29][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:20.954299] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=27][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:20.954323] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=18][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:20.954337] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:20.954347] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-04-22 16:46:20.954361] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=11][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:20.954373] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:20.954420] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=40][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:20.954464] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=36][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:20.954482] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=16][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:20.954492] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=10][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:20.954513] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345E-0-0] [lt=9][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:20.954540] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13614][ServerGTimer][T0][YB42AC175883-000616AB0807345E-0-0] [lt=22][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-04-22 16:46:20.954558] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-04-22 16:46:20.954569] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:20.954578] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:20.954589] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7e008050, start=1713775580953402, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-04-22 16:46:20.954604] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:20.954803] WDIAG [SHARE] refresh (ob_all_server_tracer.cpp:568) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-5019] fail to get servers_info(ret=-5019, ret="OB_TABLE_NOT_EXIST", GCTX.sql_proxy_=0x5608c1b14388) [2024-04-22 16:46:20.954824] WDIAG [SHARE] runTimerTask (ob_all_server_tracer.cpp:626) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] fail to refresh all server map(ret=-5019) [2024-04-22 16:46:20.955230] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=70][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:20.955300] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=50] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-04-22 16:46:20.955338] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=33][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, ret=-5019) [2024-04-22 16:46:20.955361] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=21][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_stat, db_name=oceanbase) [2024-04-22 16:46:20.955381] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_stat) [2024-04-22 16:46:20.955400] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:20.955412] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:20.955430] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_sys_stat' doesn't exist [2024-04-22 16:46:20.955442] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:20.955454] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:20.955465] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:20.955475] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:20.955533] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=55][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:20.955554] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=18][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:20.955573] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=16][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:20.955603] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=19][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:20.955617] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=12][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:20.955681] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=60][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:20.955694] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:20.955705] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2024-04-22 16:46:20.955720] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:20.955732] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:20.955758] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:20.955808] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=41][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:20.955822] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=13][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:20.955832] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:20.955855] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:20.955883] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB0897339F-0-0] [lt=25][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, aret=-5019, ret=-5019) [2024-04-22 16:46:20.955901] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [14014][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2024-04-22 16:46:20.955916] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [14014][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:20.955928] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [14014][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:20.955941] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [14014][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f4dbdf7a050, start=1713775580954983, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2024-04-22 16:46:20.955959] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [14014][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:20.955972] WDIAG [SHARE] fetch_max_id (ob_max_id_fetcher.cpp:479) [14014][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute sql failed(sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2024-04-22 16:46:20.956069] WDIAG [SQL.QRR] fetch_max_rule_version (ob_udr_sql_service.cpp:141) [14014][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-5019] failed to fetch max rule version(ret=-5019, tenant_id=1) [2024-04-22 16:46:20.956104] WDIAG [SQL.QRR] sync_rule_from_inner_table (ob_udr_mgr.cpp:252) [14014][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-5019] failed to fetch max rule version(ret=-5019) [2024-04-22 16:46:20.956125] WDIAG [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:95) [14014][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] failed to sync rule from inner table(ret=-5019) [2024-04-22 16:46:20.958384] WDIAG load_file_to_string (utility.h:627) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:46:20.958433] WDIAG get_ethernet_speed (utility.cpp:580) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:46:20.958455] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:46:20.958476] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:46:20.972170] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:20.972240] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=25000, remain_us=1594946, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:20.984793] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:21.001497] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.001593] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=26000, remain_us=1565593, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.012168] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.012209] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=20669719, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=138, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:21.021552] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:237) [13917][T1_Occam][T1][YB42AC175883-000616AB08273813-0-0] [lt=27] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-22 16:46:21.021761] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:237) [13917][T1_Occam][T1][YB42AC175883-000616AB08273815-0-0] [lt=47] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-22 16:46:21.031812] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.031916] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=27000, remain_us=1535270, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.044384] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:131) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=18] ====== checkpoint timer task ====== [2024-04-22 16:46:21.044470] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:748) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=52] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2024-04-22 16:46:21.044528] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1023) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=45] get_min_unreplayed_log_info(lsn={lsn:1373666}, scn={val:1713775580944599173, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713775580958476, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:7851, ret_code_:0}}, next_to_submit_lsn_:{lsn:1373666}, next_to_submit_scn_:{val:1713775580944599173, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f4d7c89f000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:1373544}, end_lsn:{lsn:1373666}, read_buf:{buf_len_:8192, buf_:0x7f4d7c89f000}, block_size:67104768, log_storage_:0x7f4d7e49b4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713775580944599172, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713775580944599172, v:0}, data_checksum:2923900223, flag:0}}, init_mode_version:0, accumulate_checksum:407953226, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713775580944599172, v:0}}}}}) [2024-04-22 16:46:21.046709] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13787][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.047309] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13801][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.047399] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13806][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.047528] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13779][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.047580] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1426) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=179] succ to get rec scn(*this={this:0x7f4d7c604030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1713773681663296, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:46:21.047682] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:180) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=99] tx ctx memtable get rec scn(this={ObITable:{this:0x7f4d7e532080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f4d7e532080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:46:21.047757] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:856) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=62] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-04-22 16:46:21.047764] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13802][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.047757] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13783][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.047884] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13793][BatchIO][T0][Y0-0000000000000000-0-0] [lt=86] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.047942] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13799][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=62] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.047953] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13794][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.047837] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:259) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=26] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f4d7e454510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713775597210958129, latest_log_ts:{val:1713775580545744419, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x7f4d953f4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2024-04-22 16:46:21.048000] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:599) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=156] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1}) [2024-04-22 16:46:21.048028] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:470) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=17] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2024-04-22 16:46:21.048058] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=24] get rec log scn(service_type_=0, rec_log_ts={val:1713775580545744419, v:0}) [2024-04-22 16:46:21.048076] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=17] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-04-22 16:46:21.048094] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] get rec log scn(service_type_=2, rec_log_ts={val:1713773717210958135, v:0}) [2024-04-22 16:46:21.048115] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:164) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713773717210958135, v:0}, checkpoint_scn_in_ls_meta={val:1713773717210958135, v:0}, ls_id={id:1}, service_type="DAS_ID") [2024-04-22 16:46:21.048151] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1579) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.23.88.131:2882", has_set_deleted:false}, curr_base_lsn={lsn:0}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2024-04-22 16:46:21.048199] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:185) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=46] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2024-04-22 16:46:21.048235] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:192) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=28] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-04-22 16:46:21.048441] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13792][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.048511] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13797][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=35] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.048568] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13788][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.048707] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13800][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=33] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.048724] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13791][BatchIO][T0][Y0-0000000000000000-0-0] [lt=36] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.048745] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13790][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.048746] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13804][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.048908] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13780][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.048913] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13798][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.049214] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13782][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.049416] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13785][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.049545] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13784][RpcIO][T0][Y0-0000000000000000-0-0] [lt=51] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.049708] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13796][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=41] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.049784] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=17][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:21.049835] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=32] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:46:21.049911] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=70][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:46:21.049941] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=27][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:46:21.049973] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=29][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:46:21.049996] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=17][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:21.050009] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:21.050026] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:46:21.050038] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:21.050085] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=43][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:21.050107] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=20][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:21.050124] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=15][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:21.050139] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13789][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.050142] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=16][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:21.050160] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=16][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:21.050179] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=17][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:21.050205] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=14][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:21.050220] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:21.050234] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:21.050314] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=76][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:21.050335] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=18][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:46:21.050354] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=16][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:21.050309] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13781][RpcIO][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:21.050381] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=24][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.050410] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=23][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:21.050437] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=22][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:21.050448] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:21.050458] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:21.050481] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973975-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:21.050501] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973975-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:46:21.050526] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973975-0-0] [lt=22][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:21.050556] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973975-0-0] [lt=26][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:21.050615] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973975-0-0] [lt=53][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:21.050675] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973975-0-0] [lt=56][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7f58a050, start=1713775581049384, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:21.050704] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973975-0-0] [lt=27][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:21.050754] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973975-0-0] [lt=44][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:46:21.050869] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973975-0-0] [lt=26][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:46:21.050899] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973975-0-0] [lt=28][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:46:21.063538] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.063681] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=28000, remain_us=1503507, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.095775] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.095903] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=29000, remain_us=1471284, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.116722] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.116834] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=20565095, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=139, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:21.122046] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=77] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:46:21.122473] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:46:21.122512] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:21.122567] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775580944599172, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775581122463) [2024-04-22 16:46:21.122607] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775580922356, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:21.128313] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.128363] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=30000, remain_us=1438823, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.139498] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [14059][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=16] ====== check clog disk timer task ====== [2024-04-22 16:46:21.139557] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [14059][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=50] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=65) [2024-04-22 16:46:21.139700] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [14059][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=58] cannot_recycle_log_size statistics(cannot_recycle_log_size=1373666, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:46:21.147579] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=33] Cache replace map node details(ret=0, replace_node_count=0, replace_time=5272, replace_start_pos=2453646, replace_num=62914) [2024-04-22 16:46:21.147611] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:46:21.158395] INFO [LIB] stat (utility.h:1165) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=120] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=5032, this=0x7f4d7e49f330, extra_info="palf_id:1") [2024-04-22 16:46:21.158461] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=62] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.131:2882", has_set_deleted:false}, accum_size=860) [2024-04-22 16:46:21.158543] INFO [LIB] stat (utility.h:1182) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=50] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=78, this=0x7f4d95343700, extra_info="") [2024-04-22 16:46:21.158715] INFO [LIB] stat (utility.h:1165) [13927][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=47] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=91, this=0x7f4d7e49f560, extra_info="palf_id:1") [2024-04-22 16:46:21.159831] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E4E-0-0] [lt=7] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=19, this=0x7f4d7e4956a0, extra_info="palf_id:1") [2024-04-22 16:46:21.159859] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E4E-0-0] [lt=27] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=6340, this=0x7f4d7e4958d0, extra_info="palf_id:1") [2024-04-22 16:46:21.159868] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E4E-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.131:2882", total_slide_log_cnt=7, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=17, avg_log_submit_to_flush_time=5201, avg_log_submit_to_slide_time=6323) [2024-04-22 16:46:21.159902] INFO [PALF] read (log_cache.cpp:100) [14081][T1_ReplaySrv2][T1][Y0-0000000000000000-0-0] [lt=52] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=860, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2024-04-22 16:46:21.160030] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1172) [14081][T1_ReplaySrv2][T1][Y0-0000000000000000-0-0] [lt=94] succ to submit log task to replay service(to_submit_lsn={lsn:1373722}, to_submit_scn={val:1713775581153075552, v:0}, replay_status={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713775581159818, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:7851, ret_code_:0}}, next_to_submit_lsn_:{lsn:1373788}, next_to_submit_scn_:{val:1713775581153075553, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f4d7c89f000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:1373666}, end_lsn:{lsn:1373788}, read_buf:{buf_len_:8192, buf_:0x7f4d7c89f000}, block_size:67104768, log_storage_:0x7f4d7e49b4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713775581153075552, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713775581153075552, v:0}, data_checksum:2923900223, flag:1}}, init_mode_version:0, accumulate_checksum:3029363489, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713775581153075552, v:0}}}}}) [2024-04-22 16:46:21.160284] INFO [LIB] stat (utility.h:1165) [14083][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=55] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=128, this=0x7f4d98c048f8, extra_info="") [2024-04-22 16:46:21.160344] INFO [LIB] stat (utility.h:1165) [14083][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=57] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=44, this=0x7f4d98c04b28, extra_info="") [2024-04-22 16:46:21.161832] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [13692][LSSysTblUp0][T0][YB42AC175883-000616AB08A9F612-0-0] [lt=29][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713775570633812}) [2024-04-22 16:46:21.162175] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.162210] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=31000, remain_us=1404975, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.162526] INFO eloop_run (eloop.c:140) [13667][pnio1][T0][Y0-0000000000000000-0-0] [lt=55] PNIO [ratelimit] time: 1713775581162522, bytes: 91538869, bw: 0.026428 MB/s, add_ts: 1007635, add_bytes: 27923 [2024-04-22 16:46:21.185479] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=77] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:21.193234] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=11] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:18, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713775573499769, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"]) [2024-04-22 16:46:21.195288] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=145][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:21.195382] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=75] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:46:21.195434] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=47][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:46:21.195475] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=38][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:46:21.195516] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=38][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:46:21.195558] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=34][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:21.195593] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=32][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:21.195614] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=13][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:46:21.195679] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=61][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:21.195701] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=20][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:21.195732] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=28][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:21.195746] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=13][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:21.195780] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=25][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:21.195793] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=11][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:21.195812] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=16][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:21.195857] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=32][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:21.195891] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=29][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:21.195930] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=34][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:21.195964] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=31][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:21.195980] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=14][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:46:21.196014] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=30][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:21.196048] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=30][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.196096] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=41][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:21.196146] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=42][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:21.196190] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=41][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:21.196205] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=14][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:21.196249] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=30][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:21.196289] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=37][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:46:21.196315] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=17][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:21.196331] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=14][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:21.196362] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=27][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:21.196386] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=21][errcode=-5019] query failed(ret=-5019, conn=0x7f4da0e3e050, start=1713775581195085, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:21.196425] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=36][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:21.196441] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:21.196552] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.196554] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=32][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:46:21.196586] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=32000, remain_us=1370599, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=32, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.196593] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=35][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:46:21.196618] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=23][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:21.196690] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=69][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:21.196727] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=31][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:46:21.196765] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=33][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:46:21.196798] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392D-0-0] [lt=31][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:21.219799] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.219839] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=20462090, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=140, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:21.219865] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [13720][SysLocAsyncUp0][T0][YB42AC175883-000616AB07685424-0-0] [lt=52][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713775581219783}) [2024-04-22 16:46:21.222561] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:193) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48] get wrs ts(ls_id={id:1}, delta=69274, log_service_ts=1713775581153075, min_tx_service_ts=4611686018427387, timestamp={val:1713775581153075552, v:0}) [2024-04-22 16:46:21.222591] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=99, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1713775577733204341, v:0}", last_lsn={lsn:1370919}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-04-22 16:46:21.222659] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2644) [13932][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26] [PALF_DUMP](palf_id=1, self="172.23.88.131:2882", [SlidingWindow]={palf_id:1, self:"172.23.88.131:2882", lsn_allocator:{max_log_id:11271, max_lsn:1373788, max_scn:{val:1713775581153075552, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:1373788}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:0}}, last_submit_lsn:{lsn:1373666}, last_submit_end_lsn:{lsn:1373788}, last_submit_log_id:11271, last_submit_log_pid:18, max_flushed_lsn:{lsn:1373666}, max_flushed_end_lsn:{lsn:1373788}, max_flushed_log_pid:18, committed_end_lsn:{lsn:1373788}, last_slide_log_id:11271, last_slide_scn:{val:1713775581153075552, v:0}, last_slide_lsn:{lsn:1373666}, last_slide_end_lsn:{lsn:1373788}, last_slide_log_pid:18, last_slide_log_accum_checksum:3029363489, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f4d7e494530}, [StateMgr]={this:0x7f4d7e498330, self:"172.23.88.131:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:18}, leader:"172.23.88.129:2882", leader_epoch:28, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:11272, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x7f4d7e49ebf0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.23.88.131:2882", alive_paxos_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:18, prev:{version:1, config:{config_version:{proposal_id:17, config_seq:18}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:18, config_seq:19}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:17, prev_lsn:{lsn:1368258}, prev_mode_pid:18}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:18, config_seq:19}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", paxos_member_region_map:{}, register_time_us:1713775579220057, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1713775579220057, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f4d7e495c70}, [ModeMgr]={palf_id:1, self:"172.23.88.131:2882", applied_mode_meta:{version:1, proposal_id:18, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:18, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:18, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:18}, log_config_meta:{version:3, proposal_id:18, prev:{version:1, config:{config_version:{proposal_id:17, config_seq:18}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:18, config_seq:19}, log_sync_memberlist:3{server:"172.23.88.129:2882", timestamp:1, flag:0}{server:"172.23.88.130:2882", timestamp:1, flag:0}{server:"172.23.88.131:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:17, prev_lsn:{lsn:1368258}, prev_mode_pid:18}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:18, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:233472}, readable_log_tail:{lsn:233472}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/meta", dir_fd:237, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66871296, block_header_serialize_buf_:0x7f4d7e49a4b8, flashback_version:0}, log_storage:{log_tail:{lsn:1373788}, readable_log_tail:{lsn:1373788}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/creality/oceanbase/store/clog/tenant_1/1/log", dir_fd:238, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:65730980, block_header_serialize_buf_:0x7f4d7e49c138, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1713775152657955, this:0x7f4d7e498a70}, [Reconfirm]={palf_id:1, self:"172.23.88.131:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f4d7e4984b0}) [2024-04-22 16:46:21.223181] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2741) [13932][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=512][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="172.23.88.131:2882", tmp_server={server:"172.23.88.129:2882", timestamp:1, flag:0}) [2024-04-22 16:46:21.223237] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2648) [13932][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=53] [PALF_DUMP](palf_id=1, self="172.23.88.131:2882", ack_info_list=[]) [2024-04-22 16:46:21.223287] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2741) [13932][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=43][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="172.23.88.131:2882", tmp_server={server:"172.23.88.129:2882", timestamp:1, flag:0}) [2024-04-22 16:46:21.223346] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4287) [13932][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=56] current log_task status(palf_id=1, self="172.23.88.131:2882", log_id=11272, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2024-04-22 16:46:21.225697] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.225727] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.225738] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.225746] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.225751] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.225764] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.225768] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.225772] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.225776] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.225781] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.225785] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.225789] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.225794] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.225798] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.225805] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.225809] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:21.233257] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.233345] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=33000, remain_us=1333842, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=33, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.234059] INFO eloop_run (eloop.c:140) [13674][pnio2][T0][Y0-0000000000000000-0-0] [lt=62] PNIO [ratelimit] time: 1713775581234054, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007567, add_bytes: 0 [2024-04-22 16:46:21.236340] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC1A-0-0] [lt=530][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:21.236419] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC1A-0-0] [lt=75][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:21.236492] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC1A-0-0] [lt=69][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:21.236553] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC1A-0-0] [lt=58][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:21.270379] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.270480] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=34000, remain_us=1296707, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=34, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.309103] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.309207] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=35000, remain_us=1257980, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=35, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.322666] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:21.322730] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=60][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775581153075552, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775581322652) [2024-04-22 16:46:21.322759] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775581122656, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:21.323792] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.323929] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=20358000, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=141, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:21.347796] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=44] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=9) [2024-04-22 16:46:21.348112] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.348223] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=36000, remain_us=1218963, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=36, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.386187] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=71] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20809842688, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:21.387967] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.388089] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=37000, remain_us=1179098, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=37, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.427480] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.427557] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=20254372, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=142, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:21.429799] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.429857] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=38000, remain_us=1137329, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=38, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.431372] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [13622][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:595, tid:13685}]) [2024-04-22 16:46:21.468000] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:38, local_retry_times:38, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:21.468094] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.468124] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.468138] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:21.468164] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:21.473417] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=21][errcode=-4725] [2024-04-22 16:46:21.473470] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=50][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:21.473503] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:21.473544] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=38][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:21.473560] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:21.473572] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:21.473584] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.473702] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.473758] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=47][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.473781] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:21.473802] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=39, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.473837] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20] will sleep(sleep_us=39000, remain_us=1093349, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=39, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.512960] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:39, local_retry_times:39, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:21.513019] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=44][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.513055] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=34][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.513069] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:21.513101] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:21.517900] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] [2024-04-22 16:46:21.517937] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=34][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:21.517966] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:21.517985] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:21.518000] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:21.518012] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:21.518023] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.518065] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.518112] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=41][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.518149] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=34][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:21.518171] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=40, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.518204] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19] will sleep(sleep_us=40000, remain_us=1048982, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=40, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.522849] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:21.522893] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775581352951089, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775581522839) [2024-04-22 16:46:21.522916] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775581322793, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:21.527689] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=22][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:142, local_retry_times:142, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:21.527791] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=82][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.527819] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.527856] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=35][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:21.527883] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:21.533137] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] [2024-04-22 16:46:21.533192] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=53][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:21.533219] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:21.533236] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:21.533250] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:21.533262] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:21.533273] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.533329] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.533368] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.533382] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:21.533432] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=46][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=143, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.533471] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=24] will sleep(sleep_us=100000, remain_us=20148458, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=143, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:21.538387] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [13778][TsMgr][T0][Y0-0000000000000000-0-0] [lt=66] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:46:21.540135] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=40][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:21.540203] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=57] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:21.540247] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=27] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:46:21.540293] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=39][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:46:21.540355] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=59][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:46:21.540388] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=31][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:46:21.540417] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=19] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:21.540436] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=16][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:21.540456] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=17][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:21.540522] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=53][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:46:21.540540] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=15][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:21.540559] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=17][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:21.540576] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=15][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:21.540593] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=15][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:21.540611] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=16][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:21.540681] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=65][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:21.540714] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=29][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:21.540771] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=39][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:21.540796] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=20][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:21.540821] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=19][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:21.540840] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=17][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:21.540857] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=15][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-04-22 16:46:21.540944] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=69][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:21.540972] WDIAG [SERVER] do_process (obmp_query.cpp:760) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=26][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-04-22 16:46:21.540995] WDIAG [SERVER] do_process (obmp_query.cpp:884) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=16][errcode=-5019] query failed(ret=-5019, session={this:0x7f4d7edd01f8, id:3221225477, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-04-22 16:46:21.541049] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673524-0-0] [lt=30] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0x579c808 0x578fbfb 0x5788b5f 0x577a048 0xabd78a4 0x16c398be 0x7f4dc8ea7ea5 0x7f4dc8bd0b0d") [2024-04-22 16:46:21.547943] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=57] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=8) [2024-04-22 16:46:21.558361] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:40, local_retry_times:40, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:21.558467] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=81][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.558519] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=47][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.558560] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=38][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:21.558611] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=33][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:21.562502] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=100][errcode=-4725] [2024-04-22 16:46:21.562529] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:21.562564] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=33][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:21.562575] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:21.562588] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:21.562611] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=21][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:21.562623] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.562680] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=34] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.562709] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.562720] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:21.562741] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=41, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.562765] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13] will sleep(sleep_us=41000, remain_us=1004421, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=41, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.586984] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=91] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:21.603918] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:41, local_retry_times:41, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:21.604035] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=91][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.604066] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=28][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.604084] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:21.604135] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=21][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:21.607747] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=41][errcode=-4725] [2024-04-22 16:46:21.607807] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=57][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:21.607850] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=40][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:21.607875] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:21.607891] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:21.607903] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:21.607914] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.607969] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.608030] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=54][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.608051] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=17][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:21.608083] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=29][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=42, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.608130] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=27] will sleep(sleep_us=42000, remain_us=959057, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=42, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.622975] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:21.623101] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=78][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775581352951089, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775581622961) [2024-04-22 16:46:21.623198] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=92][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775581522936, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:21.623272] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=60] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:46:21.633713] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=36][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:143, local_retry_times:143, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:21.633802] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=67][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.633823] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.633835] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:21.633884] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:21.637755] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=45][errcode=-4725] [2024-04-22 16:46:21.637809] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=51][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:21.637842] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:21.637861] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:21.637876] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:21.637887] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:21.637898] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.637945] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.637999] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=46][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.638022] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:21.638052] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=27][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=144, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.638117] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=49] will sleep(sleep_us=100000, remain_us=20043812, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=144, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:21.650272] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=22][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:42, local_retry_times:42, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:21.650347] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=56][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.650365] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=17][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.650375] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:21.650398] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:21.654806] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] [2024-04-22 16:46:21.654843] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=35][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:21.654882] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=36][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:21.654895] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:21.654908] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:21.654918] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:21.654928] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.654971] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=8] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.655019] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=41][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.655033] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:21.655049] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=43, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.655086] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=22] will sleep(sleep_us=43000, remain_us=912101, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=43, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.698220] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=17][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:43, local_retry_times:43, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:21.698391] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=148][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.698414] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=22][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.698426] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:21.698453] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:21.698569] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2192) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19] need check schema(new_schema_version=1, cached_tenant_schema_version=1, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = ? ORDER BY row_id, column_name) [2024-04-22 16:46:21.702421] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=94][errcode=-4725] [2024-04-22 16:46:21.702461] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=38][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:21.702491] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:21.702509] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:21.702523] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:21.702535] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:21.702565] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=28][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.702618] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.702701] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=76][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.702714] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:21.702732] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=44, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.702765] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18] will sleep(sleep_us=44000, remain_us=864421, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=44, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.732762] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC1D-0-0] [lt=321][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:21.732830] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC1D-0-0] [lt=65][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:21.732913] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC1D-0-0] [lt=79][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:21.732972] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC1D-0-0] [lt=55][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:21.738270] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=35][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:144, local_retry_times:144, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:21.738356] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.738378] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.738390] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:21.738417] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:21.742432] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=28][errcode=-4725] [2024-04-22 16:46:21.742483] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=49][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:21.742517] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:21.742535] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:21.742550] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:21.742562] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:21.742607] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=42][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.742700] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.742768] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=60][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.742785] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=14][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:21.742803] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=145, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.742838] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19] will sleep(sleep_us=100000, remain_us=19939091, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=145, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:21.746918] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=29][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:44, local_retry_times:44, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:21.747034] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=94][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.747081] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=44][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.747103] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:21.747141] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:21.748083] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=7) [2024-04-22 16:46:21.750055] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=26][errcode=-4725] [2024-04-22 16:46:21.750099] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=41][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:21.750129] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:21.750148] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:21.750161] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:21.750172] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:21.750183] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.750231] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.750289] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=50][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.750310] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=17][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:21.750345] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=45, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.750433] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=70] will sleep(sleep_us=45000, remain_us=816754, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=45, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.787759] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=87] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:21.795571] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:45, local_retry_times:45, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:21.795650] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=41][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.795672] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.795680] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=7][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:21.795698] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=8][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:21.799157] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] [2024-04-22 16:46:21.799187] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=28][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:21.799209] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:21.799228] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:21.799241] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:21.799267] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:21.799278] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.799313] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=6] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.799337] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.799345] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=6][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:21.799356] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=46, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.799377] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10] will sleep(sleep_us=46000, remain_us=767809, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=46, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.823165] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:21.823231] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775581548216799, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775581823153) [2024-04-22 16:46:21.823260] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775581623259, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:21.842968] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:145, local_retry_times:145, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:21.843053] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=60][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.843069] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=15][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.843078] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=8][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:21.843136] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=26][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:21.845538] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:46, local_retry_times:46, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:21.845739] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=176][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.845769] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=28][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.845789] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:21.845865] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=44][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:21.846615] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] [2024-04-22 16:46:21.846662] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=45][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:21.846679] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:21.846693] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:21.846703] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:21.846710] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=6][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:21.846717] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=7][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.846747] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=6] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.846775] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=23][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.846786] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:21.846806] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=146, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.846852] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=31] will sleep(sleep_us=100000, remain_us=19835077, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=146, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:21.848560] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=47][errcode=-4725] [2024-04-22 16:46:21.848607] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=44][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:21.848668] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=58][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:21.848685] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:21.848699] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:21.848711] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:21.848722] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.848767] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.848821] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=45][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:21.848843] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:21.848872] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=26][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=47, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.848936] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=43] will sleep(sleep_us=47000, remain_us=718250, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=47, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.896115] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:47, local_retry_times:47, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:21.896247] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=103][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.896277] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:21.896292] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:21.896322] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:21.900096] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=30][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:21.900192] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.900269] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=48000, remain_us=666917, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=48, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.924761] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=32][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:21.924832] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=52] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:46:21.924868] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=31][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:46:21.924889] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=19][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:46:21.924906] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:46:21.924923] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:21.924935] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:21.924950] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:46:21.924962] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:21.925007] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=43][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:21.925020] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=11][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:21.925030] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:21.925040] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=9][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:21.925051] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:21.925062] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:21.925084] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:21.925097] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:21.925109] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:21.925120] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:21.925131] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='172.23.88.131' and svr_port=2882, ret=-5019) [2024-04-22 16:46:21.925144] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=11][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:21.925155] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.23.88.131' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.925188] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=27][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:21.925213] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=20][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:21.925224] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:21.925233] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:21.925262] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=9][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.23.88.131' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:21.925277] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A1-0-0] [lt=14][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.23.88.131' and svr_port=2882"}, aret=-5019, ret=-5019) [2024-04-22 16:46:21.925291] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='172.23.88.131' and svr_port=2882) [2024-04-22 16:46:21.925303] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:21.925315] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:21.925326] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7c0fc050, start=1713775581924532, sql=SELECT zone FROM __all_server where svr_ip='172.23.88.131' and svr_port=2882) [2024-04-22 16:46:21.925342] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:21.925361] WDIAG get_my_sql_result_ (ob_table_access_helper.h:435) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f4d94a56298, table=__all_server, condition=where svr_ip='172.23.88.131' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='172.23.88.131' and svr_port=2882, columns_str="zone") [2024-04-22 16:46:21.925389] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:332) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='172.23.88.131' and svr_port=2882) [2024-04-22 16:46:21.925480] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:635) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f4d94a56298, where_condition="where svr_ip='172.23.88.131' and svr_port=2882", zone_name_holder=) [2024-04-22 16:46:21.925516] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:568) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-04-22 16:46:21.925540] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:577) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-04-22 16:46:21.925555] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:144) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:21.936429] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=32][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:21.936530] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=75] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:46:21.936582] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=48][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-04-22 16:46:21.936612] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=27][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-04-22 16:46:21.936689] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=74][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-04-22 16:46:21.936721] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=23][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:21.936743] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=20][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:21.936769] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=17][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-04-22 16:46:21.936794] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=22][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:21.936807] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=12][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:21.936859] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=49][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:21.936880] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:4307) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=19][errcode=-5019] resolve table failed(ret=-5019) [2024-04-22 16:46:21.936904] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:3867) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=21][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-04-22 16:46:21.936923] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3723) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=18][errcode=-5019] resolve joined table failed(ret=-5019) [2024-04-22 16:46:21.936940] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=15][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:21.936957] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=16][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:21.936983] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=22][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:21.937028] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=41][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:21.937066] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=24][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:21.937099] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=28][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:21.937138] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=33][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:21.937173] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=31][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:21.937203] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=24][errcode=-5019] fail to handle text query(stmt=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-04-22 16:46:21.937273] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=65][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:21.937296] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=20][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:21.937357] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=53][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:21.937423] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=55][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:21.937443] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:21.937459] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=14][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:21.937492] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349D-0-0] [lt=17][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:21.937533] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [14114][BlackListServic][T0][YB42AC175883-000616AB07E7349D-0-0] [lt=36][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-04-22 16:46:21.937569] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:46:21.937592] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:21.937674] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:21.937711] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7c0e8050, start=1713775581935997, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:46:21.937736] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:21.937749] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:219) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:46:21.937863] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:193) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=15] ls blacklist refresh finish(cost_time=4134) [2024-04-22 16:46:21.937983] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [14062][T1_LSArchiveMgr][T1][YB42AC175883-000616AB07773224-0-0] [lt=44] gc stale ls task succ [2024-04-22 16:46:21.938381] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [13752][Blacklist][T0][Y0-0000000000000000-0-0] [lt=48] blacklist_loop exec finished(cost_time=39, is_enabled=true, send_cnt=0) [2024-04-22 16:46:21.939534] INFO do_work (ob_rl_mgr.cpp:704) [13683][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=48] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:46:21.940810] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [14055][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=75] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=13101, mem_used=68631552) [2024-04-22 16:46:21.941672] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [14072][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=45] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:46:21.947017] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=17][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:21.948269] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=54] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=6) [2024-04-22 16:46:21.952063] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.952175] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=19729755, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=147, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:21.952674] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:21.952709] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=49000, remain_us=614476, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=49, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:21.953984] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF786-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=0, this=0x7f4d7e495470, extra_info="") [2024-04-22 16:46:21.954068] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF786-0-0] [lt=57] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.131:2882", role="FOLLOWER", total_group_log_cnt=5, avg_log_batch_cnt=0, total_group_log_size=601, avg_group_log_size=120) [2024-04-22 16:46:21.954109] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF786-0-0] [lt=35] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.131:2882", committed size=601) [2024-04-22 16:46:21.958168] INFO pn_ratelimit (group.c:589) [13817][IngressService][T0][Y0-0000000000000000-0-0] [lt=48] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-04-22 16:46:21.958789] WDIAG load_file_to_string (utility.h:627) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:46:21.958825] WDIAG get_ethernet_speed (utility.cpp:580) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:46:21.958844] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:46:21.958857] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:46:21.988472] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=68] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:22.005456] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.005542] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=50000, remain_us=561644, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=50, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:22.023439] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:22.023546] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=102][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775581753885611, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775582023412) [2024-04-22 16:46:22.023594] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775581823290, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:22.047242] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13793][BatchIO][T0][Y0-0000000000000000-0-0] [lt=73] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.047313] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13806][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=66] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.047719] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13779][RpcIO][T0][Y0-0000000000000000-0-0] [lt=60] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.047861] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13787][BatchIO][T0][Y0-0000000000000000-0-0] [lt=52] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.047903] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13802][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.047923] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13792][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.047969] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13783][RpcIO][T0][Y0-0000000000000000-0-0] [lt=111] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048044] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13794][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048073] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13799][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=29] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048105] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13791][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048152] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13790][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048471] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13801][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048674] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13788][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048665] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13797][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048830] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13800][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=37] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048840] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13796][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=41] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048865] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13798][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=40] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048956] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13782][RpcIO][T0][Y0-0000000000000000-0-0] [lt=46] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048948] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13804][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=82] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.049038] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13780][RpcIO][T0][Y0-0000000000000000-0-0] [lt=42] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.049546] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13785][RpcIO][T0][Y0-0000000000000000-0-0] [lt=48] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.049697] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13784][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.050042] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13789][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.050471] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13781][RpcIO][T0][Y0-0000000000000000-0-0] [lt=75] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.053048] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=17][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:22.053131] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=59] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:46:22.053165] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:46:22.053185] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:46:22.053206] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=19][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:46:22.053224] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:22.053236] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:22.053255] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=11][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:46:22.053268] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:22.053431] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=156][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:22.053457] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=24][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:22.053468] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:22.053478] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:22.053489] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:22.053499] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:22.053529] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:22.053542] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:22.053562] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=15][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:22.053577] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=13][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:22.053591] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=11][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:46:22.053699] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=97][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:22.053717] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=15][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.053784] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=58][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:22.053835] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=43][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:22.053850] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=14][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:22.053860] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:22.053885] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=12][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:22.053912] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=22][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:46:22.053949] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=32][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:22.053970] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=18][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:22.053992] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=20][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:22.054004] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7c05c050, start=1713775582052785, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:22.054030] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=24][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:22.054041] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=7][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:46:22.054159] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=11][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:46:22.054193] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=31][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:46:22.054798] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.054917] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=19627013, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=148, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:22.059087] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.059138] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=51000, remain_us=508048, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=51, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:22.114414] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.114538] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=52000, remain_us=452649, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=52, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:22.123334] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:46:22.148473] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=51] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=5) [2024-04-22 16:46:22.157251] INFO pktc_sk_new (pktc_sk_factory.h:78) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO sk_new: s=0x7f4d7e667358 [2024-04-22 16:46:22.157460] INFO pktc_do_connect (pktc_post.h:19) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=49] PNIO sk_new: sk=0x7f4d7e667358, fd=243 [2024-04-22 16:46:22.157523] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=35] [ussl] write client fd succ, fd:243, gid:0x100000006, need_send_negotiation:1 [2024-04-22 16:46:22.157565] INFO eloop_regist (eloop.c:47) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=41] PNIO sock regist: 0x7f4d7e667358 fd=243 [2024-04-22 16:46:22.157587] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO sock not ready: 0x7f4d7e667358, fd=243 [2024-04-22 16:46:22.157665] INFO ussl_eloop_regist (ussl_eloop.c:39) [13666][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] [ussl] sock regist: 0x7f4daa015bd0 fd=243 [2024-04-22 16:46:22.157905] INFO handle_client_writable_event (handle-event.c:157) [13666][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=59] [ussl] client send negotiation message succ, fd:243, addr:172.23.88.131:50194, auth_method:NONE, gid:0x100000006 [2024-04-22 16:46:22.158010] INFO clientfd_sk_delete (ussl_factory.c:52) [13666][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=52] [ussl] give back fd to origin epoll succ, client_fd:243, client_epfd:145, event:0x8000000d, client_addr:172.23.88.130:2882, has_error:0 [2024-04-22 16:46:22.158148] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO sock connect OK: 0x7f4d7e667358 fd:243:local:172.23.88.131:50194:remote:172.23.88.130:2882 [2024-04-22 16:46:22.159455] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.159578] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=19522352, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=149, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:22.159899] INFO [LIB] stat (utility.h:1165) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT WRITE LOG COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=5572, this=0x7f4d7e49f330, extra_info="palf_id:1") [2024-04-22 16:46:22.159977] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=75] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.131:2882", has_set_deleted:false}, accum_size=601) [2024-04-22 16:46:22.160044] INFO [LIB] stat (utility.h:1182) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=72, this=0x7f4d95343700, extra_info="") [2024-04-22 16:46:22.160168] INFO [LIB] stat (utility.h:1165) [13927][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=87] [PALF STAT FLUSH CB COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=83, this=0x7f4d7e49f560, extra_info="palf_id:1") [2024-04-22 16:46:22.161286] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E53-0-0] [lt=20] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=31, this=0x7f4d7e4956a0, extra_info="palf_id:1") [2024-04-22 16:46:22.161348] INFO [PALF] read (log_cache.cpp:100) [14083][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=601, hit_cnt=5, read_cnt=5, hit rate=1.000000000000000000e+00) [2024-04-22 16:46:22.161387] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E53-0-0] [lt=99] [PALF STAT LOG LIFE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=6847, this=0x7f4d7e4958d0, extra_info="palf_id:1") [2024-04-22 16:46:22.161411] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E53-0-0] [lt=22] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.131:2882", total_slide_log_cnt=5, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=24, avg_log_submit_to_flush_time=5758, avg_log_submit_to_slide_time=6823) [2024-04-22 16:46:22.161512] INFO [LIB] stat (utility.h:1165) [14082][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=28] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=101, this=0x7f4d98c048f8, extra_info="") [2024-04-22 16:46:22.161580] INFO [LIB] stat (utility.h:1165) [14082][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=66] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=10, this=0x7f4d98c04b28, extra_info="") [2024-04-22 16:46:22.166965] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [13692][LSSysTblUp0][T0][YB42AC175883-000616AB08A9F675-0-0] [lt=34][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713775570633812}) [2024-04-22 16:46:22.170136] INFO eloop_run (eloop.c:140) [13667][pnio1][T0][Y0-0000000000000000-0-0] [lt=63] PNIO [ratelimit] time: 1713775582170131, bytes: 91558305, bw: 0.018396 MB/s, add_ts: 1007609, add_bytes: 19436 [2024-04-22 16:46:22.170601] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.170736] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=53000, remain_us=396451, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=53, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:22.189152] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=80] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:22.196741] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] server is initiating(server_id=0, local_seq=1064, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:46:22.196812] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=64] new mysql sessid created(conn.sessid_=3221226536, support_ssl=false) [2024-04-22 16:46:22.196921] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221226536, sess.client_addr_="127.0.0.1:49348") [2024-04-22 16:46:22.196973] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1019) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=37] accept one succ(*s={this:0x7f4d73604030, session_id:3221226536, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:244}, err:0, last_decode_time:0, last_write_time:1713775582196914, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:46:22.197201] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=102] username and tenantname(user_name=ocp_monitor, tenant_name=) [2024-04-22 16:46:22.197288] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [13815][sql_nio5][T1][Y0-0000000000000000-0-0] [lt=40] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-04-22 16:46:22.197397] INFO [SERVER] verify_connection (obmp_connect.cpp:1964) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=35] server is initializing, ignore verify_ip_white_list(status=1, ret=0) [2024-04-22 16:46:22.197739] INFO load_privilege_info (obmp_connect.cpp:545) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=57] no tenant name set, use default tenant name(tenant_name=sys) [2024-04-22 16:46:22.198048] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=14] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:18, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713775573499769, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"]) [2024-04-22 16:46:22.198900] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=43] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:22.198957] INFO [SHARE.SCHEMA] get_user_info (ob_schema_getter_guard.cpp:2213) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=45] user not exist(tenant_id=1, user_name=ocp_monitor) [2024-04-22 16:46:22.198977] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2620) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=17][errcode=-4043] No tenant user(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"127.0.0.1", db:"oceanbase", scramble_str:"GUS#SYG7{*Zi!dTZH:(W"}, ret=-4043, ret="OB_PASSWORD_WRONG") [2024-04-22 16:46:22.199023] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:678) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=42][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"127.0.0.1", db:"oceanbase", scramble_str:"GUS#SYG7{*Zi!dTZH:(W"}, ret=-4043) [2024-04-22 16:46:22.199045] WDIAG [SERVER] verify_identify (obmp_connect.cpp:2062) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=18][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-04-22 16:46:22.199063] WDIAG [SERVER] process (obmp_connect.cpp:346) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=15][errcode=-4043] fail to verify_identify(ret=-4043) [2024-04-22 16:46:22.199098] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=11] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0xaec6a47 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f4dc8ea7ea5 0x7f4dc8bd0b0d") [2024-04-22 16:46:22.199207] INFO [SERVER] free_session (obmp_base.cpp:326) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=19] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221226536, proxy_sessid:0}) [2024-04-22 16:46:22.199257] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:835) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=37][errcode=0] server close connection(sessid=3221226536, proxy_sessid=0, stack="0x5d1c83c 0xaeeb5ca 0xaeb47cd 0xaec6183 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f4dc8ea7ea5 0x7f4dc8bd0b0d") [2024-04-22 16:46:22.199278] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221226536, proxy_sessid=0) [2024-04-22 16:46:22.199293] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:839) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=14][errcode=-4016] session is null [2024-04-22 16:46:22.199308] INFO [SERVER] process (obmp_connect.cpp:488) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=xxx.xxx.xxx.xxx, sessid=3221226536, proxy_sessid=0, client_sessid=3221226536, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-04-22 16:46:22.199369] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=55][errcode=-4018] get session fail(ret=-4018, sessid=3221226536, proxy_sessid=0) [2024-04-22 16:46:22.199386] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1068) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=15][errcode=-4018] fail to get session info(ret=-4018) [2024-04-22 16:46:22.199432] WDIAG [SERVER] response (obmp_base.cpp:89) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=21][errcode=-4018] failed to flush_buffer(ret=-4018) [2024-04-22 16:46:22.199448] WDIAG [RPC.FRAME] run (ob_sql_processor.cpp:48) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=14][errcode=-4018] response rpc result fail(ret=-4018) [2024-04-22 16:46:22.199527] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:825) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4015] close sql sock by user req(*s={this:0x7f4d73604030, session_id:3221226536, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:244}, err:5, last_decode_time:1713775582197195, last_write_time:1713775582199524, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:101, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:46:22.199576] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=107][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:22.199597] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=58] kill and revert session(conn.sessid_=3221226536, proxy_sessid=0, server_id=0, ret=0) [2024-04-22 16:46:22.199621] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=28] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:46:22.199619] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:915) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=20] can close safely, do destroy(*s={this:0x7f4d73604030, session_id:3221226536, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:244}, err:5, last_decode_time:1713775582197195, last_write_time:1713775582199524, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:101, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-04-22 16:46:22.199682] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=56][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:46:22.199698] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=14][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:46:22.199684] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=60] connection close(sessid=3221226536, proxy_sessid=0, tenant_id=1, server_id=0, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:89, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-04-22 16:46:22.199713] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:46:22.199746] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=27][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:22.199756] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=9][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:22.199769] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:46:22.199778] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:22.199787] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:22.199795] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=7][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:22.199803] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:22.199812] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:22.199821] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:22.199829] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=7][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:22.199856] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=17][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:22.199866] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:22.199877] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:22.199886] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:22.199894] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=6][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:46:22.199906] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=10][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:22.199915] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.199935] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=16][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:22.199956] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=16][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:22.199964] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:22.199971] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=7][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:22.199997] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:22.200015] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=17][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:46:22.200026] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:22.200036] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=9][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:22.200046] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:22.200056] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7e02c050, start=1713775582199393, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:22.200069] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=12][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:22.200079] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=7][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:22.200151] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=10][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:46:22.200163] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=10][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:46:22.200173] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:22.223125] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [13953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=57] ====== tenant freeze timer task ====== [2024-04-22 16:46:22.223695] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:46:22.223777] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=74][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:22.223885] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=69][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775581953044062, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775582223681) [2024-04-22 16:46:22.223915] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [13720][SysLocAsyncUp0][T0][YB42AC175883-000616AB07685444-0-0] [lt=71][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1713775582223849}) [2024-04-22 16:46:22.223933] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775582023694, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:22.227505] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227537] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227551] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227707] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=149] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227738] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227750] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227726] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.227768] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227783] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227792] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227801] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227811] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227820] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227797] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=54000, remain_us=339389, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=54, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:22.227829] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227849] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227859] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227868] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.233606] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [13622][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=74][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:5, tid:14043}]) [2024-04-22 16:46:22.239036] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC20-0-0] [lt=280][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:22.239104] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC20-0-0] [lt=64][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:22.239173] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC20-0-0] [lt=64][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:22.239231] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC20-0-0] [lt=55][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:22.241672] INFO eloop_run (eloop.c:140) [13674][pnio2][T0][Y0-0000000000000000-0-0] [lt=58] PNIO [ratelimit] time: 1713775582241669, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2024-04-22 16:46:22.263232] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.263332] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=19418597, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=150, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:22.286029] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.286151] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=55000, remain_us=281036, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=55, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:22.023439] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:22.023546] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=102][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775581753885611, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775582023412) [2024-04-22 16:46:22.023594] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775581823290, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:22.047242] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13793][BatchIO][T0][Y0-0000000000000000-0-0] [lt=73] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.047313] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13806][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=66] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.047719] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13779][RpcIO][T0][Y0-0000000000000000-0-0] [lt=60] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.047861] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13787][BatchIO][T0][Y0-0000000000000000-0-0] [lt=52] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.047903] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13802][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.047923] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13792][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.047969] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13783][RpcIO][T0][Y0-0000000000000000-0-0] [lt=111] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048044] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13794][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048073] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13799][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=29] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048105] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13791][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048152] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13790][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048471] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13801][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048674] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13788][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048665] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13797][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048830] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13800][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=37] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048840] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13796][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=41] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048865] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13798][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=40] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048956] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13782][RpcIO][T0][Y0-0000000000000000-0-0] [lt=46] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.048948] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13804][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=82] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.049038] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13780][RpcIO][T0][Y0-0000000000000000-0-0] [lt=42] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.049546] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13785][RpcIO][T0][Y0-0000000000000000-0-0] [lt=48] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.049697] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13784][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.050042] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13789][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.050471] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13781][RpcIO][T0][Y0-0000000000000000-0-0] [lt=75] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:22.053048] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=17][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:22.053131] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=59] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:46:22.053165] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:46:22.053185] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:46:22.053206] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=19][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:46:22.053224] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:22.053236] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:22.053255] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=11][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:46:22.053268] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:22.053431] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=156][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:22.053457] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=24][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:22.053468] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:22.053478] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:22.053489] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:22.053499] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:22.053529] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:22.053542] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:22.053562] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=15][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:22.053577] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=13][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:22.053591] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=11][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:46:22.053699] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=97][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:22.053717] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=15][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.053784] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=58][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:22.053835] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=43][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:22.053850] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=14][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:22.053860] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:22.053885] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973976-0-0] [lt=12][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:22.053912] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=22][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:46:22.053949] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=32][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:22.053970] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=18][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:22.053992] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=20][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:22.054004] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7c05c050, start=1713775582052785, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:22.054030] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=24][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:22.054041] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=7][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:46:22.054159] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=11][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:46:22.054193] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973976-0-0] [lt=31][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:46:22.054798] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.054917] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=19627013, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=148, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:22.059087] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.059138] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=51000, remain_us=508048, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=51, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:22.114414] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.114538] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=52000, remain_us=452649, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=52, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:22.123334] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:46:22.148473] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=51] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=5) [2024-04-22 16:46:22.157251] INFO pktc_sk_new (pktc_sk_factory.h:78) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO sk_new: s=0x7f4d7e667358 [2024-04-22 16:46:22.157460] INFO pktc_do_connect (pktc_post.h:19) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=49] PNIO sk_new: sk=0x7f4d7e667358, fd=243 [2024-04-22 16:46:22.157523] INFO ussl_loop_add_clientfd (ussl-loop.c:250) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=35] [ussl] write client fd succ, fd:243, gid:0x100000006, need_send_negotiation:1 [2024-04-22 16:46:22.157565] INFO eloop_regist (eloop.c:47) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=41] PNIO sock regist: 0x7f4d7e667358 fd=243 [2024-04-22 16:46:22.157587] INFO pktc_sk_check_connect (pktc_sk_factory.h:17) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO sock not ready: 0x7f4d7e667358, fd=243 [2024-04-22 16:46:22.157665] INFO ussl_eloop_regist (ussl_eloop.c:39) [13666][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] [ussl] sock regist: 0x7f4daa015bd0 fd=243 [2024-04-22 16:46:22.157905] INFO handle_client_writable_event (handle-event.c:157) [13666][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=59] [ussl] client send negotiation message succ, fd:243, addr:172.23.88.131:50194, auth_method:NONE, gid:0x100000006 [2024-04-22 16:46:22.158010] INFO clientfd_sk_delete (ussl_factory.c:52) [13666][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=52] [ussl] give back fd to origin epoll succ, client_fd:243, client_epfd:145, event:0x8000000d, client_addr:172.23.88.130:2882, has_error:0 [2024-04-22 16:46:22.158148] INFO pktc_sk_check_connect (pktc_sk_factory.h:25) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO sock connect OK: 0x7f4d7e667358 fd:243:local:172.23.88.131:50194:remote:172.23.88.130:2882 [2024-04-22 16:46:22.159455] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.159578] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=19522352, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=149, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:22.159899] INFO [LIB] stat (utility.h:1165) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT WRITE LOG COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=5572, this=0x7f4d7e49f330, extra_info="palf_id:1") [2024-04-22 16:46:22.159977] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=75] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.131:2882", has_set_deleted:false}, accum_size=601) [2024-04-22 16:46:22.160044] INFO [LIB] stat (utility.h:1182) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=38] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=72, this=0x7f4d95343700, extra_info="") [2024-04-22 16:46:22.160168] INFO [LIB] stat (utility.h:1165) [13927][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=87] [PALF STAT FLUSH CB COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=83, this=0x7f4d7e49f560, extra_info="palf_id:1") [2024-04-22 16:46:22.161286] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E53-0-0] [lt=20] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=31, this=0x7f4d7e4956a0, extra_info="palf_id:1") [2024-04-22 16:46:22.161348] INFO [PALF] read (log_cache.cpp:100) [14083][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=601, hit_cnt=5, read_cnt=5, hit rate=1.000000000000000000e+00) [2024-04-22 16:46:22.161387] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E53-0-0] [lt=99] [PALF STAT LOG LIFE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=6847, this=0x7f4d7e4958d0, extra_info="palf_id:1") [2024-04-22 16:46:22.161411] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E53-0-0] [lt=22] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.131:2882", total_slide_log_cnt=5, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=24, avg_log_submit_to_flush_time=5758, avg_log_submit_to_slide_time=6823) [2024-04-22 16:46:22.161512] INFO [LIB] stat (utility.h:1165) [14082][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=28] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=101, this=0x7f4d98c048f8, extra_info="") [2024-04-22 16:46:22.161580] INFO [LIB] stat (utility.h:1165) [14082][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=66] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=10, this=0x7f4d98c04b28, extra_info="") [2024-04-22 16:46:22.166965] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [13692][LSSysTblUp0][T0][YB42AC175883-000616AB08A9F675-0-0] [lt=34][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713775570633812}) [2024-04-22 16:46:22.170136] INFO eloop_run (eloop.c:140) [13667][pnio1][T0][Y0-0000000000000000-0-0] [lt=63] PNIO [ratelimit] time: 1713775582170131, bytes: 91558305, bw: 0.018396 MB/s, add_ts: 1007609, add_bytes: 19436 [2024-04-22 16:46:22.170601] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.170736] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=53000, remain_us=396451, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=53, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:22.189152] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=80] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:22.196741] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] server is initiating(server_id=0, local_seq=1064, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:46:22.196812] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=64] new mysql sessid created(conn.sessid_=3221226536, support_ssl=false) [2024-04-22 16:46:22.196921] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221226536, sess.client_addr_="127.0.0.1:49348") [2024-04-22 16:46:22.196973] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1019) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=37] accept one succ(*s={this:0x7f4d73604030, session_id:3221226536, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:244}, err:0, last_decode_time:0, last_write_time:1713775582196914, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:46:22.197201] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=102] username and tenantname(user_name=ocp_monitor, tenant_name=) [2024-04-22 16:46:22.197288] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [13815][sql_nio5][T1][Y0-0000000000000000-0-0] [lt=40] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-04-22 16:46:22.197397] INFO [SERVER] verify_connection (obmp_connect.cpp:1964) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=35] server is initializing, ignore verify_ip_white_list(status=1, ret=0) [2024-04-22 16:46:22.197739] INFO load_privilege_info (obmp_connect.cpp:545) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=57] no tenant name set, use default tenant name(tenant_name=sys) [2024-04-22 16:46:22.198048] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=14] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:18, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713775573499769, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"]) [2024-04-22 16:46:22.198900] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=43] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:22.198957] INFO [SHARE.SCHEMA] get_user_info (ob_schema_getter_guard.cpp:2213) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=45] user not exist(tenant_id=1, user_name=ocp_monitor) [2024-04-22 16:46:22.198977] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2620) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=17][errcode=-4043] No tenant user(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"127.0.0.1", db:"oceanbase", scramble_str:"GUS#SYG7{*Zi!dTZH:(W"}, ret=-4043, ret="OB_PASSWORD_WRONG") [2024-04-22 16:46:22.199023] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:678) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=42][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"127.0.0.1", db:"oceanbase", scramble_str:"GUS#SYG7{*Zi!dTZH:(W"}, ret=-4043) [2024-04-22 16:46:22.199045] WDIAG [SERVER] verify_identify (obmp_connect.cpp:2062) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=18][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-04-22 16:46:22.199063] WDIAG [SERVER] process (obmp_connect.cpp:346) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=15][errcode=-4043] fail to verify_identify(ret=-4043) [2024-04-22 16:46:22.199098] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=11] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0xaec6a47 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f4dc8ea7ea5 0x7f4dc8bd0b0d") [2024-04-22 16:46:22.199207] INFO [SERVER] free_session (obmp_base.cpp:326) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=19] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221226536, proxy_sessid:0}) [2024-04-22 16:46:22.199257] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:835) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=37][errcode=0] server close connection(sessid=3221226536, proxy_sessid=0, stack="0x5d1c83c 0xaeeb5ca 0xaeb47cd 0xaec6183 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f4dc8ea7ea5 0x7f4dc8bd0b0d") [2024-04-22 16:46:22.199278] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221226536, proxy_sessid=0) [2024-04-22 16:46:22.199293] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:839) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=14][errcode=-4016] session is null [2024-04-22 16:46:22.199308] INFO [SERVER] process (obmp_connect.cpp:488) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=12] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=xxx.xxx.xxx.xxx, sessid=3221226536, proxy_sessid=0, client_sessid=3221226536, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-04-22 16:46:22.199369] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=55][errcode=-4018] get session fail(ret=-4018, sessid=3221226536, proxy_sessid=0) [2024-04-22 16:46:22.199386] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1068) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=15][errcode=-4018] fail to get session info(ret=-4018) [2024-04-22 16:46:22.199432] WDIAG [SERVER] response (obmp_base.cpp:89) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=21][errcode=-4018] failed to flush_buffer(ret=-4018) [2024-04-22 16:46:22.199448] WDIAG [RPC.FRAME] run (ob_sql_processor.cpp:48) [14047][T1_MysqlQueueTh][T1][Y0-000616AB09073384-0-0] [lt=14][errcode=-4018] response rpc result fail(ret=-4018) [2024-04-22 16:46:22.199527] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:825) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4015] close sql sock by user req(*s={this:0x7f4d73604030, session_id:3221226536, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:244}, err:5, last_decode_time:1713775582197195, last_write_time:1713775582199524, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:101, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:46:22.199576] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=107][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:22.199597] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=58] kill and revert session(conn.sessid_=3221226536, proxy_sessid=0, server_id=0, ret=0) [2024-04-22 16:46:22.199621] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=28] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:46:22.199619] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:915) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=20] can close safely, do destroy(*s={this:0x7f4d73604030, session_id:3221226536, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:244}, err:5, last_decode_time:1713775582197195, last_write_time:1713775582199524, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:101, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-04-22 16:46:22.199682] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=56][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:46:22.199698] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=14][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:46:22.199684] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [13815][sql_nio5][T0][Y0-0000000000000000-0-0] [lt=60] connection close(sessid=3221226536, proxy_sessid=0, tenant_id=1, server_id=0, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:89, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-04-22 16:46:22.199713] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:46:22.199746] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=27][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:22.199756] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=9][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:22.199769] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:46:22.199778] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:22.199787] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:22.199795] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=7][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:22.199803] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:22.199812] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:22.199821] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:22.199829] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=7][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:22.199856] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=17][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:22.199866] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:22.199877] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:22.199886] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:22.199894] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=6][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:46:22.199906] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=10][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:22.199915] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.199935] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=16][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:22.199956] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=16][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:22.199964] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:22.199971] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=7][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:22.199997] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:22.200015] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=17][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:46:22.200026] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:22.200036] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=9][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:22.200046] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:22.200056] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=8][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7e02c050, start=1713775582199393, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:22.200069] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=12][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:22.200079] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=7][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:22.200151] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=10][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:46:22.200163] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=10][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:46:22.200173] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392E-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:22.223125] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [13953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=57] ====== tenant freeze timer task ====== [2024-04-22 16:46:22.223695] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:46:22.223777] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=74][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:22.223885] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=69][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775581953044062, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775582223681) [2024-04-22 16:46:22.223915] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [13720][SysLocAsyncUp0][T0][YB42AC175883-000616AB07685444-0-0] [lt=71][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1713775582223849}) [2024-04-22 16:46:22.223933] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775582023694, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:22.227505] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227537] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227551] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227707] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=149] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227738] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227750] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227726] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.227768] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227783] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227792] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227801] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227811] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227820] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227797] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=54000, remain_us=339389, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=54, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:22.227829] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227849] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227859] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.227868] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:22.233606] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [13622][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=74][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:5, tid:14043}]) [2024-04-22 16:46:22.239036] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC20-0-0] [lt=280][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:22.239104] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC20-0-0] [lt=64][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:22.239173] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC20-0-0] [lt=64][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:22.239231] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC20-0-0] [lt=55][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:22.241672] INFO eloop_run (eloop.c:140) [13674][pnio2][T0][Y0-0000000000000000-0-0] [lt=58] PNIO [ratelimit] time: 1713775582241669, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2024-04-22 16:46:22.263232] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.263332] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=19418597, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=150, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:22.286029] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.286151] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=55000, remain_us=281036, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=55, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:22.345085] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.345180] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=56000, remain_us=222006, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=56, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:22.348880] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=259] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=4) [2024-04-22 16:46:22.367744] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.367869] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=19314061, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=151, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:22.389734] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=90] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20809842688, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:22.405169] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.405319] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=57000, remain_us=161867, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=57, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:22.423926] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:22.424032] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=100][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775582249272637, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775582423907) [2024-04-22 16:46:22.424077] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775582223957, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:22.466321] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.466413] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=58000, remain_us=100773, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=58, v.err_=-4725, timeout_timestamp=1713775582567185) [2024-04-22 16:46:22.473816] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.473927] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=19208002, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=152, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:22.529397] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=2][errcode=-4012] [2024-04-22 16:46:22.529443] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=44][errcode=-4012] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4012, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:22.529494] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=48][errcode=-4012] failed to setup first scanner(ret=-4012) [2024-04-22 16:46:22.529511] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4012] Open this operator failed(ret=-4012, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:22.529527] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4012] open operator failed(ret=-4012) [2024-04-22 16:46:22.529539] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4012] fail open main query(ret=-4012) [2024-04-22 16:46:22.529551] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:46:22.529562] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:46:22.529573] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:22.529594] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=59, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.529681] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=83][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:59, local_retry_times:59, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-04-22 16:46:22.529718] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:46:22.529731] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:46:22.529749] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=17][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-04-22 16:46:22.529794] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=24][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-04-22 16:46:22.529814] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=16] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1960908) [2024-04-22 16:46:22.529830] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-04-22 16:46:22.529847] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:46:22.529862] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-04-22 16:46:22.529874] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:22.529888] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-04-22 16:46:22.529912] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:46:22.530013] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-04-22 16:46:22.530036] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=21][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:46:22.530049] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4012] get failed(ret=-4012) [2024-04-22 16:46:22.530062] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:888) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:46:22.530081] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4507) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=17][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:46:22.530097] WDIAG [SHARE.SCHEMA] operator() (ob_multi_version_schema_service.cpp:2425) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4012] fail to get baseline_schema_version(ret=-4012) [2024-04-22 16:46:22.530110] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2495) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=11] [REFRESH_SCHEMA] end refresh and add schema(ret=-4012, ret="OB_TIMEOUT", tenant_ids=[], cost=1962943) [2024-04-22 16:46:22.530128] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:308) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=17][errcode=-4012] fail to refresh and add schema(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:46:22.530140] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:314) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=11] [REFRESH_SCHEMA] end refresh schema with new mode, (ret=-4012, ret="OB_TIMEOUT", tenant_ids=[], used time=1963039) [2024-04-22 16:46:22.534539] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [13622][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=126][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:255, tid:13685}]) [2024-04-22 16:46:22.536345] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] [2024-04-22 16:46:22.536400] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=52][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:22.536433] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:22.536458] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:22.536501] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=41][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:22.536516] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:22.536527] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.536578] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=8] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.536741] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=152][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.536831] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=85][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:22.536919] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=85][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.536965] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=28] already timeout, do not need sleep(sleep_us=0, remain_us=1993205, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.536989] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:22.537073] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.537092] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.537143] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=36][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:22.537192] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=32][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:22.539337] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=23][errcode=-4725] [2024-04-22 16:46:22.539372] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=33][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:22.539399] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=24][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:22.539414] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:22.539427] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:22.539438] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:22.539448] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.539489] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.539523] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=28][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.539535] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:22.539568] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.539598] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18] will sleep(sleep_us=1000, remain_us=1990572, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.540729] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:22.540783] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=38][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.540804] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.540816] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:22.540852] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:22.541450] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [13778][TsMgr][T0][Y0-0000000000000000-0-0] [lt=49] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=150, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-04-22 16:46:22.541509] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [13778][TsMgr][T0][Y0-0000000000000000-0-0] [lt=55] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:46:22.542175] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [14021][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=99] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:46:22.542916] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] [2024-04-22 16:46:22.542951] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=32][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:22.542971] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:22.542999] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:22.543013] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:22.543024] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:22.543034] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.543076] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.543123] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=41][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.543138] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:22.543158] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.543198] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=26] will sleep(sleep_us=2000, remain_us=1986972, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.544448] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=17][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:22.544518] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=59] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:22.544553] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=22] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:46:22.544578] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=21][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:46:22.544596] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=16][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:46:22.544621] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=23][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:46:22.544687] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=59] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:22.544715] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=25][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:22.544729] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=13][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:22.544748] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:46:22.544760] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:22.544771] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:22.544782] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:22.544793] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:22.544805] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:22.544817] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:22.544829] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:22.544852] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:22.544866] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:22.544881] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:22.544902] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=19][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:22.544914] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-04-22 16:46:22.544942] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:22.544964] WDIAG [SERVER] do_process (obmp_query.cpp:760) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=20][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-04-22 16:46:22.544984] WDIAG [SERVER] do_process (obmp_query.cpp:884) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=14][errcode=-5019] query failed(ret=-5019, session={this:0x7f4d7edd01f8, id:3221225477, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-04-22 16:46:22.545027] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673525-0-0] [lt=29] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0x579c808 0x578fbfb 0x5788b5f 0x577a048 0xabd78a4 0x16c398be 0x7f4dc8ea7ea5 0x7f4dc8bd0b0d") [2024-04-22 16:46:22.545455] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=26][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:22.545556] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=80][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.545684] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=124][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.545719] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=32][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:22.545748] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:22.547677] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] [2024-04-22 16:46:22.547713] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=34][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:22.547734] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:22.547748] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:22.547761] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:22.547772] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:22.547783] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.547871] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.547934] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=55][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.547968] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:22.547995] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=3, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.548051] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=38] will sleep(sleep_us=3000, remain_us=1982119, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.549092] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=78] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=3) [2024-04-22 16:46:22.551317] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=48][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:3, local_retry_times:3, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:22.551403] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=67][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.551424] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.551435] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:22.551481] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:22.553464] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=35][errcode=-4725] [2024-04-22 16:46:22.553501] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=34][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:22.553522] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:22.553553] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=28][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:22.553567] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:22.553578] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:22.553589] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.553717] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.553755] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.553768] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:22.553786] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=4, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.553816] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=17] will sleep(sleep_us=4000, remain_us=1976354, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.557955] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:4, local_retry_times:4, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:22.558052] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=78][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.558096] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=42][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.558111] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:22.558145] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:22.560576] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=27][errcode=-4725] [2024-04-22 16:46:22.560624] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=45][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:22.560688] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=61][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:22.560704] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:22.560718] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:22.560729] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:22.560739] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.560792] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.560830] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.560864] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:22.560882] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=5, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.560914] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18] will sleep(sleep_us=5000, remain_us=1969257, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.566056] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:5, local_retry_times:5, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:22.566123] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=48][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.566146] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.566158] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:22.566182] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:22.568915] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=17][errcode=-4725] [2024-04-22 16:46:22.568973] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=55][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:22.569025] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=49][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:22.569042] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:22.569058] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:22.569070] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:22.569091] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.569144] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.569187] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=35][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.569201] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:22.569221] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=6, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.569259] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19] will sleep(sleep_us=6000, remain_us=1960911, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.574100] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:152, local_retry_times:152, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:22.574182] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.574210] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.574245] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=33][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:22.574281] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:22.575420] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=21][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:6, local_retry_times:6, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:22.575530] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=87][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.575554] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=22][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.575566] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:22.575604] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:22.577355] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=28][errcode=-4725] [2024-04-22 16:46:22.577379] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=23][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:22.577396] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=15][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:22.577409] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:22.577420] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:22.577430] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:22.577437] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=7][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.577463] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=5] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.577482] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.577489] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=6][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:22.577511] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=21][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=153, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.577529] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10] will sleep(sleep_us=100000, remain_us=19104400, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=153, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:22.579452] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=76][errcode=-4725] [2024-04-22 16:46:22.579506] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=52][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:22.579540] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:22.579559] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:22.579573] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:22.579585] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:22.579596] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.579682] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.579764] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=73][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.579787] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:22.579808] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=7, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.579840] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=17] will sleep(sleep_us=7000, remain_us=1950330, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.586991] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:7, local_retry_times:7, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:22.587070] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=59][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.587096] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=23][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.587115] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:22.587143] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:22.590415] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20807745536, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:22.591312] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] [2024-04-22 16:46:22.591352] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=37][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:22.591400] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=44][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:22.591417] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:22.591430] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:22.591441] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:22.591452] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.591500] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.591569] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=60][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.591600] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=26][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:22.591679] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=76][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=8, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.591717] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=24] will sleep(sleep_us=8000, remain_us=1938454, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.599959] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:8, local_retry_times:8, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:22.600213] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=227][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.600249] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=33][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.600270] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:22.600384] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=21][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:22.604338] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=53][errcode=-4725] [2024-04-22 16:46:22.604401] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=60][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:22.604438] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=35][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:22.604459] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:22.604481] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:22.604499] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:22.604517] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=17][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.604582] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.604725] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=133][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.604753] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:22.604780] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=9, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.604823] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=27] will sleep(sleep_us=9000, remain_us=1925347, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.607590] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [13930][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=88] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):65, sequence:2}) [2024-04-22 16:46:22.613971] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:9, local_retry_times:9, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:22.614040] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=50][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.614065] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=22][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.614083] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:22.614128] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:22.617851] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] [2024-04-22 16:46:22.617905] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=52][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:22.617973] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=65][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:22.617990] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:22.618004] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:22.618015] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:22.618026] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.618084] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.618122] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.618170] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=45][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:22.618191] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=10, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.618259] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=53] will sleep(sleep_us=10000, remain_us=1911912, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.624178] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:22.624251] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=69][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775582453074359, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775582624163) [2024-04-22 16:46:22.624286] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775582424098, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:22.624323] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:46:22.628397] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=22][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:10, local_retry_times:10, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:22.628477] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=61][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.628498] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.628510] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:22.628555] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:22.632260] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=37][errcode=-4725] [2024-04-22 16:46:22.632311] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=48][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:22.632344] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:22.632362] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:22.632376] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:22.632409] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:22.632420] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.632468] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.632522] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=46][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:22.632544] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:22.632564] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=11, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:22.632597] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19] will sleep(sleep_us=11000, remain_us=1897573, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.639795] INFO keepalive_check (time_wheel.c:76) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=50] PNIO user_keepalive_timeout has been reset, sock=(ptr=0x7f4d7e667358,dest=-2108155988:2882), old=0, new=3000000 [2024-04-22 16:46:22.643842] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=105][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:11, local_retry_times:11, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:22.643957] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=95][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.643985] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:22.644020] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=33][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:22.644046] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:22.644533] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:189) [14116][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-04-22 16:46:22.644611] WDIAG [SERVER] do_heartbeat_event (ob_heartbeat.cpp:266) [14116][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=54][errcode=0] try reload schema failed(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=-4023) [2024-04-22 16:46:22.647595] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] [2024-04-22 16:46:22.647693] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=95][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:22.647721] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:22.647736] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:22.647750] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:22.647801] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.647867] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=12000, remain_us=1882303, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.664312] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.664398] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=13000, remain_us=1865772, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.681183] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.681266] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=19000663, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=154, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:22.681439] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.681530] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=14000, remain_us=1848640, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.699361] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.699445] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=15000, remain_us=1830725, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.718420] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.718527] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=16000, remain_us=1811643, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.734679] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC23-0-0] [lt=265][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:22.734747] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC23-0-0] [lt=64][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:22.734798] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC23-0-0] [lt=47][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:22.734834] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC23-0-0] [lt=34][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:22.738495] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.738603] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=17000, remain_us=1791567, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.749291] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=59] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=2) [2024-04-22 16:46:22.759537] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.759661] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=18000, remain_us=1770545, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.772275] INFO ussl_eloop_regist (ussl_eloop.c:39) [13666][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] [ussl] sock regist: 0x7f4daa015bd0 fd=245 [2024-04-22 16:46:22.772335] INFO ussl_on_accept (ussl_listenfd.c:40) [13666][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=45] [ussl] accept new connection, fd:245, src_addr:172.23.88.130:38920 [2024-04-22 16:46:22.772422] INFO acceptfd_handle_first_readable_event (handle-event.c:422) [13666][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] auth mothod is NONE, the fd will be dispatched, fd:245, src_addr:172.23.88.130:38920 [2024-04-22 16:46:22.772448] INFO dispatch_accept_fd_to_certain_group (group.c:642) [13666][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] PNIO dispatch fd to certain group, fd:245, gid:0x100000006 [2024-04-22 16:46:22.772579] INFO pkts_sk_init (pkts_sk_factory.h:23) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=62] PNIO set pkts_sk_t sock_id s=0x7f4d7cc84048, s->id=720894 [2024-04-22 16:46:22.772655] INFO pkts_sk_new (pkts_sk_factory.h:51) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=74] PNIO sk_new: s=0x7f4d7cc84048 [2024-04-22 16:46:22.772728] INFO eloop_regist (eloop.c:47) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=14] PNIO sock regist: 0x7f4d7cc84048 fd=245 [2024-04-22 16:46:22.772795] INFO on_accept (listenfd.c:39) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=39] PNIO accept new connection, ns=0x7f4d7cc84048, fd=fd:245:local:172.23.88.131:2882:remote:172.23.88.130:38920 [2024-04-22 16:46:22.772909] WDIAG listenfd_handle_event (listenfd.c:71) [13673][pnio1][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] PNIO do_accept failed, err=11, errno=11, fd=-1 [2024-04-22 16:46:22.781758] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.781838] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=19000, remain_us=1748332, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.785222] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.785312] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=18896618, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=155, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:22.790860] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=67] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20809842688, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:22.800916] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=10015, clean_start_pos=1006632, clean_num=125829) [2024-04-22 16:46:22.803958] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.804018] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=20000, remain_us=1726151, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.824418] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=72][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:22.824545] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=86][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775582553854399, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775582824401) [2024-04-22 16:46:22.824579] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775582624312, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:22.828090] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.828180] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=21000, remain_us=1701990, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.853031] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.853073] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=22000, remain_us=1677096, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.878366] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.878465] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=23000, remain_us=1651706, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.889759] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.889849] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=18792080, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=156, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:22.901673] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=33][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:22.905123] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.905206] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=24000, remain_us=1624965, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.933092] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.933209] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=25000, remain_us=1596961, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.938075] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [14062][T1_LSArchiveMgr][T1][YB42AC175883-000616AB07773224-0-0] [lt=93] gc stale ls task succ [2024-04-22 16:46:22.939690] INFO do_work (ob_rl_mgr.cpp:704) [13683][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=38] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:46:22.939737] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [14156][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=64] current monitor number(seq_=-1) [2024-04-22 16:46:22.940987] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [14055][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=41] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=1, size_used=13151, mem_used=68631552) [2024-04-22 16:46:22.941852] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [14072][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=51] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:46:22.949470] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=59] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=1) [2024-04-22 16:46:22.959311] WDIAG load_file_to_string (utility.h:627) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:46:22.959377] WDIAG get_ethernet_speed (utility.cpp:580) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:46:22.959400] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:46:22.959469] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=65][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:46:22.962205] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.962297] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=26000, remain_us=1567873, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.990029] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=32][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:22.992109] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.992233] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=27000, remain_us=1537937, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:22.994405] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:22.994518] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=18687412, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=157, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:23.001422] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20809842688, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:23.023130] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.023222] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=28000, remain_us=1506948, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.024757] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:23.024852] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=90][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775582852917971, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775583024739) [2024-04-22 16:46:23.024896] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775582824600, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:23.047044] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13802][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.047168] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13780][RpcIO][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.047410] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13793][BatchIO][T0][Y0-0000000000000000-0-0] [lt=55] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.047672] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13801][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=61] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.047759] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13806][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.047818] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13779][RpcIO][T0][Y0-0000000000000000-0-0] [lt=61] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.047895] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13797][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=52] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.048004] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13787][BatchIO][T0][Y0-0000000000000000-0-0] [lt=50] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.048129] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13799][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=26] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.048129] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13800][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=44] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.048126] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13783][RpcIO][T0][Y0-0000000000000000-0-0] [lt=82] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.048154] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13794][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.048285] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13796][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.048323] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13790][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.048687] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13798][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=51] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.048778] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13784][RpcIO][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.048805] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13788][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.048847] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13792][BatchIO][T0][Y0-0000000000000000-0-0] [lt=58] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.049022] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13804][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=34] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.049246] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13791][BatchIO][T0][Y0-0000000000000000-0-0] [lt=51] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.049592] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13781][RpcIO][T0][Y0-0000000000000000-0-0] [lt=39] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.049698] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13785][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.050064] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13782][RpcIO][T0][Y0-0000000000000000-0-0] [lt=38] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.050148] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13789][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:23.055161] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF794-0-0] [lt=23] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x7f4d7e495470, extra_info="") [2024-04-22 16:46:23.055193] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF794-0-0] [lt=20] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.131:2882", role="FOLLOWER", total_group_log_cnt=7, avg_log_batch_cnt=0, total_group_log_size=854, avg_group_log_size=122) [2024-04-22 16:46:23.055205] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF794-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.131:2882", committed size=854) [2024-04-22 16:46:23.055253] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.055352] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=29000, remain_us=1474818, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.056529] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=13][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:23.056607] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=56] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:46:23.056713] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=101][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:46:23.056738] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=23][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:46:23.056765] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=24][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:46:23.056807] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=33][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:23.056823] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=15][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:23.056848] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=16][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:46:23.056877] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=26][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:23.056906] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=27][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:23.056935] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=26][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:23.056963] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=25][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:23.056992] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=26][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:23.057034] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=38][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:23.057048] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=13][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:23.057074] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=14][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:23.057106] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=27][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:23.057140] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=29][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:23.057171] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=27][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:23.057200] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=25][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:46:23.057233] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=29][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:23.057263] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=27][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:23.057309] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=38][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:23.057357] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=40][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:23.057389] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=29][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:23.057429] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=38][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:23.057482] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973977-0-0] [lt=28][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:23.057520] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973977-0-0] [lt=33][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:46:23.057555] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973977-0-0] [lt=31][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:23.057587] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973977-0-0] [lt=29][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:23.057676] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973977-0-0] [lt=86][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:23.057710] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973977-0-0] [lt=31][errcode=-5019] query failed(ret=-5019, conn=0x7f4dbdf7a050, start=1713775583056248, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:23.057760] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973977-0-0] [lt=47][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:23.057806] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973977-0-0] [lt=40][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:46:23.057940] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973977-0-0] [lt=45][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:46:23.057986] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973977-0-0] [lt=43][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:46:23.088393] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.088501] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=30000, remain_us=1441670, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.098911] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.099002] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=18582927, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=158, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:23.122307] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.122369] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=31000, remain_us=1407800, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.124721] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=81] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:46:23.139971] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [14059][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=75] ====== check clog disk timer task ====== [2024-04-22 16:46:23.140014] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [14059][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=36] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=65) [2024-04-22 16:46:23.140052] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [14059][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=19] cannot_recycle_log_size statistics(cannot_recycle_log_size=1374999, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:46:23.149673] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=44] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:46:23.157739] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.157830] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=32000, remain_us=1372340, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=32, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.169221] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [14065][T1_ArcSender][T1][YB42AC175883-000616AB07A73224-0-0] [lt=72][errcode=-4018] try free send task failed(ret=-4018) [2024-04-22 16:46:23.174474] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [13692][LSSysTblUp0][T0][YB42AC175883-000616AB08A9F6D8-0-0] [lt=67][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713775570633812}) [2024-04-22 16:46:23.177764] INFO eloop_run (eloop.c:140) [13667][pnio1][T0][Y0-0000000000000000-0-0] [lt=68] PNIO [ratelimit] time: 1713775583177759, bytes: 91584265, bw: 0.024570 MB/s, add_ts: 1007628, add_bytes: 25960 [2024-04-22 16:46:23.193812] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.193898] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=33000, remain_us=1336273, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=33, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.201411] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=1] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:18, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713775573499769, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"]) [2024-04-22 16:46:23.202234] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=49] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20809842688, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:23.202773] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.202881] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=18479048, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=159, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:23.203320] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=133][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:23.203396] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=56] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:46:23.203459] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=57][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:46:23.203481] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=20][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:46:23.203499] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=16][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:46:23.203517] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:23.203528] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=10][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:23.203544] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:46:23.203555] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=9][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:23.203565] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:23.203576] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:23.203585] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:23.203596] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:23.203606] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:23.203661] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=52][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:23.203688] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=16][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:23.203701] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:23.203714] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:23.203725] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:23.203735] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:46:23.203749] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:23.203760] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:23.203786] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:23.203810] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:23.203820] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:23.203829] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:23.203861] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=20][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:23.203875] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=14][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:46:23.203889] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:23.203901] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:23.203912] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:23.203924] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7f58a050, start=1713775583203099, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:23.203940] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=15][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:23.203951] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:23.204038] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=13][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:46:23.204076] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=34][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:46:23.204100] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=22][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:23.204124] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=22][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:23.204138] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=10][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:46:23.204153] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=12][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:46:23.204163] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB0817392F-0-0] [lt=9][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:23.224871] INFO [PALF] check_parent_health (log_config_mgr.cpp:2764) [13932][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=72] re_register_parent reason(palf_id=1, self="172.23.88.131:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1713775579220057, last_first_register_time_us=1713773690229985, register_time_us=1713775579220057, curr_time_us=1713775583224867) [2024-04-22 16:46:23.224986] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:46:23.225023] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:23.225058] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775582852917971, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775583224972) [2024-04-22 16:46:23.225057] INFO [PALF] check_parent_health (log_config_mgr.cpp:2769) [13932][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=84] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.131:2882") [2024-04-22 16:46:23.225082] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775583024933, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:23.229443] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.229529] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=83] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.229558] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.229576] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.229589] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.229602] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.229686] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=81] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.229700] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.229713] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.229726] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.229751] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.229764] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.229777] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.229790] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.229802] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.229815] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:23.230251] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC26-0-0] [lt=322][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:23.230321] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC26-0-0] [lt=65][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:23.230370] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC26-0-0] [lt=46][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:23.230423] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC26-0-0] [lt=50][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:23.231428] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.231503] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [13720][SysLocAsyncUp0][T0][YB42AC175883-000616AB07685477-0-0] [lt=27][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713775583231396}) [2024-04-22 16:46:23.231535] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=34000, remain_us=1298635, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=34, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.249299] INFO eloop_run (eloop.c:140) [13674][pnio2][T0][Y0-0000000000000000-0-0] [lt=45] PNIO [ratelimit] time: 1713775583249295, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007626, add_bytes: 0 [2024-04-22 16:46:23.255779] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF796-0-0] [lt=5] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=32, this=0x7f4d7e4956a0, extra_info="palf_id:1") [2024-04-22 16:46:23.255854] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF796-0-0] [lt=72] [PALF STAT LOG LIFE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=39625, this=0x7f4d7e4958d0, extra_info="palf_id:1") [2024-04-22 16:46:23.255886] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF796-0-0] [lt=30] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.131:2882", total_slide_log_cnt=6, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=16, avg_log_submit_to_flush_time=5993, avg_log_submit_to_slide_time=39609) [2024-04-22 16:46:23.255877] INFO [PALF] read (log_cache.cpp:100) [14029][T1_ReplaySrv0][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=732, hit_cnt=6, read_cnt=6, hit rate=1.000000000000000000e+00) [2024-04-22 16:46:23.255939] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF796-0-0] [lt=50][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.131:2882", log_id=11282, log_task={header:{begin_lsn:{lsn:1374999}, end_lsn:{lsn:1375121}, log_id:11282, min_scn:{val:1713775583055187376, v:0}, max_scn:{val:1713775583055187376, v:0}, data_len:66, proposal_id:18, prev_lsn:{lsn:1374877}, prev_proposal_id:18, committed_end_lsn:{lsn:1374999}, data_checksum:667309872, accum_checksum:3774452549, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713775583055154, freeze_ts:1713775583055154, submit_ts:1713775583055183, flushed_ts:1713775583059469, gen_to_freeze cost time:0, gen_to_submit cost time:29, submit_to_flush cost time:4286}, fs_cb_begin_ts=1713775583255744, log_life_time=200590) [2024-04-22 16:46:23.256055] INFO [LIB] stat (utility.h:1165) [14083][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=38] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=79, this=0x7f4d98c048f8, extra_info="") [2024-04-22 16:46:23.256118] INFO [LIB] stat (utility.h:1165) [14083][T1_ReplaySrv4][T1][Y0-0000000000000000-0-0] [lt=60] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=8, this=0x7f4d98c04b28, extra_info="") [2024-04-22 16:46:23.261056] INFO [LIB] stat (utility.h:1165) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=5738, this=0x7f4d7e49f330, extra_info="palf_id:1") [2024-04-22 16:46:23.261126] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=66] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.131:2882", has_set_deleted:false}, accum_size=854) [2024-04-22 16:46:23.261205] INFO [LIB] stat (utility.h:1182) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=45] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=80, this=0x7f4d95343700, extra_info="") [2024-04-22 16:46:23.261358] INFO [LIB] stat (utility.h:1165) [13927][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=53] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=94, this=0x7f4d7e49f560, extra_info="palf_id:1") [2024-04-22 16:46:23.269019] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.269073] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=35000, remain_us=1261097, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=35, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.306547] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.306677] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=18375253, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=160, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:23.309285] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.309377] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=36000, remain_us=1220794, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=36, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.349489] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.349588] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=37000, remain_us=1180582, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=37, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.360202] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=60] Cache replace map node details(ret=0, replace_node_count=0, replace_time=10343, replace_start_pos=2516560, replace_num=62914) [2024-04-22 16:46:23.360262] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=56] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:46:23.390852] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.390945] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=38000, remain_us=1139225, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=38, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.403153] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=62] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:23.410859] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.410954] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=18270975, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=161, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:23.425192] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=97][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:23.425316] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=80][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775583249831264, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775583425172) [2024-04-22 16:46:23.425351] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775583225181, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:23.433095] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.433193] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=39000, remain_us=1096977, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=39, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.477565] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.477799] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=40000, remain_us=1052371, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=40, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.516343] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.516469] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=18165461, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=162, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:23.522917] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.523013] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=41000, remain_us=1007158, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=41, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.537113] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [13622][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=61][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:575, tid:13685}]) [2024-04-22 16:46:23.544619] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [13778][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1713775583043131], gts:1713775583044550844, latest_srr:[mts=1713775583544538]}) [2024-04-22 16:46:23.544748] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [13778][TsMgr][T0][Y0-0000000000000000-0-0] [lt=124] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:46:23.545551] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [13672][pnio1][T0][YB42AC175883-000616AB095740BD-0-0] [lt=36] ts response task statistics(alloc_count=14326, free_count=7162) [2024-04-22 16:46:23.547724] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=16][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:23.547787] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=51] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:23.547832] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=28] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:46:23.547860] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=25][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:46:23.547879] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:46:23.547898] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:46:23.547915] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=12] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:23.547927] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=10][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:23.547939] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:23.547958] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:46:23.547970] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:23.548002] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=30][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:23.548014] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=11][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:23.548025] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:23.548037] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:23.548049] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:23.548060] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:23.548084] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:23.548098] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:23.548114] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=13][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:23.548126] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:23.548138] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-04-22 16:46:23.548164] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=12][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:23.548189] WDIAG [SERVER] do_process (obmp_query.cpp:760) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=24][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-04-22 16:46:23.548219] WDIAG [SERVER] do_process (obmp_query.cpp:884) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=24][errcode=-5019] query failed(ret=-5019, session={this:0x7f4d7edd01f8, id:3221225477, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-04-22 16:46:23.548265] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673526-0-0] [lt=31] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0x579c808 0x578fbfb 0x5788b5f 0x577a048 0xabd78a4 0x16c398be 0x7f4dc8ea7ea5 0x7f4dc8bd0b0d") [2024-04-22 16:46:23.560421] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=9) [2024-04-22 16:46:23.564177] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=26][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:41, local_retry_times:41, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:23.564273] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=69][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.564303] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.564317] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:23.564345] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:23.569393] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=21][errcode=-4725] [2024-04-22 16:46:23.569437] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=41][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:23.569469] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:23.569510] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=37][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:23.569525] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:23.569537] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:23.569549] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.569602] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.569726] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=115][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.569749] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:23.569777] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=24][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=42, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:23.569818] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=23] will sleep(sleep_us=42000, remain_us=960353, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=42, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.603690] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=84] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:23.611970] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=40][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:42, local_retry_times:42, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:23.612057] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.612103] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=43][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.612115] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:23.612153] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:23.616290] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=30][errcode=-4725] [2024-04-22 16:46:23.616328] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=35][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:23.616356] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:23.616374] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:23.616404] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=28][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:23.616416] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:23.616429] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.616475] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.616525] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=42][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.616550] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=22][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:23.616569] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=43, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:23.616602] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18] will sleep(sleep_us=43000, remain_us=913568, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=43, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.616598] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:162, local_retry_times:162, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:23.616705] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=85][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.616723] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=17][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.616734] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:23.616760] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:23.620249] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] [2024-04-22 16:46:23.620300] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=46][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:23.620357] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=54][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:23.620373] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:23.620387] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:23.620398] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:23.620409] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.620463] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.620538] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=65][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.620569] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=27][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:23.620609] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=36][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=163, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:23.620731] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=106] will sleep(sleep_us=100000, remain_us=18061199, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=163, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:23.625422] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=135][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:23.625484] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=57][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775583354067978, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775583625406) [2024-04-22 16:46:23.625524] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775583425489, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:23.625578] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:46:23.659797] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=21][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:43, local_retry_times:43, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:23.659937] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=70][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.659970] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=29][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.659987] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:23.660024] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:23.663887] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=28][errcode=-4725] [2024-04-22 16:46:23.663941] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=45][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:23.664007] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=62][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:23.664043] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:23.664077] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=30][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:23.664092] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:23.664103] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.664153] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.664211] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=50][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.664227] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:23.664262] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=44, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:23.664316] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=37] will sleep(sleep_us=44000, remain_us=865855, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=44, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.708496] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=38][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:44, local_retry_times:44, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:23.708608] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=88][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.708691] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=80][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.708723] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=30][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:23.708769] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=17][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:23.712469] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=37][errcode=-4725] [2024-04-22 16:46:23.712512] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:23.712537] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=23][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:23.712552] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:23.712572] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:23.712590] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:23.712616] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=22][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.712755] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=94] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.712816] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=53][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.712856] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=36][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:23.712877] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=45, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:23.712911] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19] will sleep(sleep_us=45000, remain_us=817259, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=45, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.720920] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=23][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:163, local_retry_times:163, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:23.721046] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=102][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.721086] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=36][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.721101] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:23.721159] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=30][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:23.725027] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=45][errcode=-4725] [2024-04-22 16:46:23.725110] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=81][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:23.725164] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=50][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:23.725190] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=23][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:23.725212] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:23.725230] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:23.725247] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.725310] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=14] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.725455] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=136][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.725473] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=15][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:23.725499] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=22][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=164, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:23.725556] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=39] will sleep(sleep_us=100000, remain_us=17956374, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=164, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:23.736129] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC29-0-0] [lt=305][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:23.736160] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC29-0-0] [lt=29][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:23.736200] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC29-0-0] [lt=39][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:23.736216] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC29-0-0] [lt=16][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:23.758076] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:45, local_retry_times:45, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:23.758170] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=65][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.758199] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.758213] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:23.758240] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:23.760698] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=62] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=8) [2024-04-22 16:46:23.761515] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] [2024-04-22 16:46:23.761570] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=52][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:23.761693] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=118][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:23.761715] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:23.761731] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:23.761743] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:23.761754] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.761810] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.761912] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=90][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.761944] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=27][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:23.761971] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=46, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:23.762007] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19] will sleep(sleep_us=46000, remain_us=768163, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=46, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.804417] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:23.808186] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=34][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:46, local_retry_times:46, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:23.808298] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=88][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.808320] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.808332] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:23.808371] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:23.812238] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=38][errcode=-4725] [2024-04-22 16:46:23.812283] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=42][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:23.812308] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=23][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:23.812323] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:23.812338] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:23.812349] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:23.812361] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.812410] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.812503] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=84][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.812529] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=22][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:23.812555] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=24][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=47, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:23.812674] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=51] will sleep(sleep_us=47000, remain_us=717499, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=47, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.825695] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=103][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:23.825762] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=62][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775583654823522, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775583825680) [2024-04-22 16:46:23.825798] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775583625554, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:23.825816] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=119][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:164, local_retry_times:164, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:23.825883] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=48][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.825901] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=17][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.825913] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:23.825962] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:23.828119] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=57][errcode=-4725] [2024-04-22 16:46:23.828195] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=73][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:23.828228] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:23.828246] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:23.828261] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:23.828272] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:23.828283] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.828331] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.828389] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=49][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.828411] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:23.828432] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=165, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:23.828466] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19] will sleep(sleep_us=100000, remain_us=17853463, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=165, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:23.859845] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=57][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:47, local_retry_times:47, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:23.859933] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=67][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.859955] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.859966] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:23.859991] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:23.862225] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] [2024-04-22 16:46:23.862286] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=58][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:23.862402] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=111][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:23.862424] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:23.862476] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=49][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:23.862488] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:23.862499] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.862550] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.862589] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.862602] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:23.862619] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=48, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:23.862729] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=91] will sleep(sleep_us=48000, remain_us=667443, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=48, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.900876] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1918) [13951][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=17] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-04-22 16:46:23.900917] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1459) [13951][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=36] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=214748360, cache_obj_num=3, cache_node_num=3) [2024-04-22 16:46:23.900938] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1476) [13951][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=18] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=214748360, cache_obj_num=3, cache_node_num=3) [2024-04-22 16:46:23.900951] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2682) [13951][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=11] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:46:23.907755] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2401) [13951][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=3, alloc_obj_list=[{obj_id:5, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:156285}, {obj_id:99, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:175513}, {obj_id:100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:176531}]) [2024-04-22 16:46:23.907822] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2690) [13951][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=60] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:46:23.910883] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:48, local_retry_times:48, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:23.910997] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=95][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.911024] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.911045] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:23.911079] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:23.913239] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] [2024-04-22 16:46:23.913279] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=37][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:23.913307] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:23.913324] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:23.913338] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:23.913349] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:23.913360] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.913411] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.913483] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=65][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.913506] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:23.913526] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=49, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:23.913558] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18] will sleep(sleep_us=49000, remain_us=616612, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=49, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:23.928663] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=38][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:165, local_retry_times:165, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:23.928747] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.928769] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.928780] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:23.928805] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:23.932427] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] [2024-04-22 16:46:23.932469] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=39][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:23.932498] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:23.932526] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=25][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:23.932548] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:23.932559] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:23.932571] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.932675] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=31] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.932732] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=48][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.932747] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:23.932765] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=166, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:23.932797] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=18] will sleep(sleep_us=100000, remain_us=17749133, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=166, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:23.938178] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [14062][T1_LSArchiveMgr][T1][YB42AC175883-000616AB07773224-0-0] [lt=39] gc stale ls task succ [2024-04-22 16:46:23.938363] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:406) [14032][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=90] finish to resize log external storage handler(current_ts=1713775583938354, tenant_max_cpu=3, valid_ls_count=0, other_ls_count=0, new_concurrency=0) [2024-04-22 16:46:23.939834] INFO do_work (ob_rl_mgr.cpp:704) [13683][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=54] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:46:23.941197] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [14055][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=60] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=1, size_used=13188, mem_used=68631552) [2024-04-22 16:46:23.942078] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [14072][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=49] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:46:23.959889] WDIAG load_file_to_string (utility.h:627) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:46:23.959945] WDIAG get_ethernet_speed (utility.cpp:580) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=43][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:46:23.959966] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:46:23.960020] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:46:23.960868] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=57] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=7) [2024-04-22 16:46:23.962683] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=25][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:49, local_retry_times:49, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:23.962750] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=48][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.962766] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:23.962775] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=8][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:23.962797] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:23.964694] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=32][errcode=-4725] [2024-04-22 16:46:23.964727] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:23.964748] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:23.964760] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:23.964771] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:23.964779] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:23.964788] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=8][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.964823] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=7] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:23.964865] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=36][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:23.964881] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:23.964909] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=26][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=50, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:23.964964] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=43] will sleep(sleep_us=50000, remain_us=565206, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=50, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:24.005158] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=91] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:24.015109] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=29][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:50, local_retry_times:50, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:24.015200] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=67][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.015239] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=35][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.015269] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=27][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:24.015300] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:24.017749] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=34][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:24.017855] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.017940] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=51000, remain_us=512231, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=51, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:24.025031] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:237) [13917][T1_Occam][T1][YB42AC175883-000616AB08273817-0-0] [lt=31] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-22 16:46:24.025942] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:24.026040] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=63][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775583854200112, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775584025928) [2024-04-22 16:46:24.026088] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775583825820, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:24.032968] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=35][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:24.035339] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.035424] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=17646505, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=167, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:24.046255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13787][BatchIO][T0][Y0-0000000000000000-0-0] [lt=122] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.047170] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13806][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.047213] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13802][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=78] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.047542] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13793][BatchIO][T0][Y0-0000000000000000-0-0] [lt=60] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.047566] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13800][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.047802] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13801][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=45] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.047798] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13794][BatchIO][T0][Y0-0000000000000000-0-0] [lt=47] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.047981] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13797][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=40] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.047976] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13780][RpcIO][T0][Y0-0000000000000000-0-0] [lt=28] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.048063] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13779][RpcIO][T0][Y0-0000000000000000-0-0] [lt=54] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.048198] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13788][BatchIO][T0][Y0-0000000000000000-0-0] [lt=48] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.048225] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13783][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.048216] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13799][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.048250] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13785][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.048389] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13791][BatchIO][T0][Y0-0000000000000000-0-0] [lt=50] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.048689] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13790][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.048672] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13796][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.048688] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13804][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.048793] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13798][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=37] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.048912] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13792][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.049801] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13781][RpcIO][T0][Y0-0000000000000000-0-0] [lt=98] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.049875] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13784][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.050147] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13789][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.050177] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13782][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:24.056426] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7A2-0-0] [lt=76] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x7f4d7e495470, extra_info="") [2024-04-22 16:46:24.056516] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7A2-0-0] [lt=59] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.131:2882", role="FOLLOWER", total_group_log_cnt=7, avg_log_batch_cnt=0, total_group_log_size=854, avg_group_log_size=122) [2024-04-22 16:46:24.056551] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7A2-0-0] [lt=31] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.131:2882", committed size=854) [2024-04-22 16:46:24.059231] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=13][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:24.059280] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=34] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:46:24.059305] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=21][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:46:24.059318] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=12][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:46:24.059344] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=24][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:46:24.059357] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=8][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:24.059364] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=7][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:24.059375] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=6][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:46:24.059383] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=6][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:24.059390] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=6][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:24.059396] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=6][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:24.059403] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=6][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:24.059410] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=6][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:24.059417] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=6][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:24.059424] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=6][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:24.059440] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=8][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:24.059449] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=7][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:24.059465] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=13][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:24.059476] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:24.059486] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:46:24.059499] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=11][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:24.059510] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.059532] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=17][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:24.059554] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:24.059565] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:24.059575] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:24.059602] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973978-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:24.059614] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973978-0-0] [lt=10][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:46:24.059652] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973978-0-0] [lt=35][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:24.059661] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973978-0-0] [lt=8][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:24.059669] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973978-0-0] [lt=7][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:24.059678] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973978-0-0] [lt=7][errcode=-5019] query failed(ret=-5019, conn=0x7f4da0e3e050, start=1713775584059063, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:24.059689] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973978-0-0] [lt=10][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:24.059697] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973978-0-0] [lt=6][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:46:24.059762] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973978-0-0] [lt=8][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:46:24.059769] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [13620][ConfigMgr][T0][YB42AC175883-000616AB09973978-0-0] [lt=7][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:46:24.073479] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.073572] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=52000, remain_us=456599, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=52, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:24.126022] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [13932][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26] LogLoopThread round_cost_time(us)(round_cost_time=27) [2024-04-22 16:46:24.126053] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:540) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=72] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1713775583954871037, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=171177, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1713775584126048, force_self_check=false, weak_read_refresh_interval=100000) [2024-04-22 16:46:24.126072] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=76] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:46:24.127731] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.127770] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=53000, remain_us=402399, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=53, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:24.137745] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.137840] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=17544089, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=168, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:24.161055] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=59] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=6) [2024-04-22 16:46:24.179974] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [13693][LSMetaTblUp0][T0][YB42AC175883-000616AB08B9F7AB-0-0] [lt=74][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713775570633812}) [2024-04-22 16:46:24.182825] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.182872] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=54000, remain_us=347298, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=54, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:24.185381] INFO eloop_run (eloop.c:140) [13667][pnio1][T0][Y0-0000000000000000-0-0] [lt=74] PNIO [ratelimit] time: 1713775584185377, bytes: 91607305, bw: 0.021807 MB/s, add_ts: 1007618, add_bytes: 23040 [2024-04-22 16:46:24.205456] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=9] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:18, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713775573499769, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"]) [2024-04-22 16:46:24.205993] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=88] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:24.207588] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=148][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:24.207725] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=111] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:46:24.207788] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=58][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:46:24.207808] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:46:24.207828] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:46:24.207847] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:24.207860] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:24.207877] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:46:24.207890] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:24.207901] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:24.207912] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:24.207923] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:24.207935] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:24.207946] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:24.207969] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=20][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:24.207994] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:24.208008] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:24.208063] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=51][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:24.208076] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:24.208088] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:46:24.208103] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:24.208116] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.208144] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=22][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:24.208171] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=22][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:24.208182] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:24.208192] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:24.208228] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=24][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:24.208245] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=16][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:46:24.208260] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:24.208273] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=12][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:24.208285] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:24.208299] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7c0fc050, start=1713775584207377, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:24.208316] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:24.208329] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:24.208485] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=14][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:46:24.208507] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=20][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:46:24.208535] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=26][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:24.208549] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=12][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:24.208564] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=10][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:46:24.208579] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=12][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:46:24.208590] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173930-0-0] [lt=10][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:24.225407] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [13953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=65] ====== tenant freeze timer task ====== [2024-04-22 16:46:24.225528] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [14073][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=140] tx gc loop thread is running(MTL_ID()=1) [2024-04-22 16:46:24.225565] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [14073][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=35] try gc retain ctx [2024-04-22 16:46:24.226178] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=62][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:46:24.226229] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:24.226264] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775584055634721, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775584226166) [2024-04-22 16:46:24.226286] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775584026161, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:24.228807] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:253) [14073][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713775597210958129, latest_log_ts:{val:1713775580545744419, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x7f4d953f4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:46:24.231277] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231340] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=59] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231366] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231385] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231428] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231441] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231453] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231465] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231426] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC2C-0-0] [lt=130][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:24.231477] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231489] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231501] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231512] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231491] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC2C-0-0] [lt=62][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:24.231524] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231544] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231556] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231539] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC2C-0-0] [lt=45][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:24.231568] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:24.231574] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC2C-0-0] [lt=34][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:24.240790] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.240844] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [13720][SysLocAsyncUp0][T0][YB42AC175883-000616AB07685497-0-0] [lt=48][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713775584240754}) [2024-04-22 16:46:24.240884] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=55000, remain_us=289287, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=55, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:24.242691] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.242772] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=17439157, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=169, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:24.256927] INFO eloop_run (eloop.c:140) [13674][pnio2][T0][Y0-0000000000000000-0-0] [lt=57] PNIO [ratelimit] time: 1713775584256921, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007626, add_bytes: 0 [2024-04-22 16:46:24.279291] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=38][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:24.279417] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=105] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:46:24.279451] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=30][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:46:24.279471] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:46:24.279490] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:46:24.279509] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:24.279521] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:24.279538] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:46:24.279550] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:24.279577] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=25][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:24.279590] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=12][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:46:24.279601] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=10][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:46:24.279614] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:46:24.279671] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=45][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:24.279685] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=12][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:24.279700] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:24.279712] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:24.279725] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584277018), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:46:09.241985, ballot number:27', 'HAPPENED_TIME', '2024-04-22 16:46:09.264468', '', '', '172.23.88.131', 2882), ret=-5019) [2024-04-22 16:46:24.279741] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=14][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:24.279754] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584277018), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:46:09.241985, ballot number:27', 'HAPPENED_TIME', '2024-04-22 16:46:09.264468', '', '', '172.23.88.131', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.279791] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=31][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:24.279819] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=22][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:24.279831] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:24.279840] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:24.279863] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13747][Occam][T1][YB42AC175883-000616AB08F733ED-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584277018), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:46:09.241985, ballot number:27', 'HAPPENED_TIME', '2024-04-22 16:46:09.264468', '', '', '172.23.88.131', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:24.279882] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13747][Occam][T0][YB42AC175883-000616AB08F733ED-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584277018), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:46:09.241985, ballot number:27', 'HAPPENED_TIME', '2024-04-22 16:46:09.264468', '', '', '172.23.88.131', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:46:24.279897] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [13747][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584277018), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:46:09.241985, ballot number:27', 'HAPPENED_TIME', '2024-04-22 16:46:09.264468', '', '', '172.23.88.131', 2882)) [2024-04-22 16:46:24.279937] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13747][Occam][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:24.279959] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [13747][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584277018), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:46:09.241985, ballot number:27', 'HAPPENED_TIME', '2024-04-22 16:46:09.264468', '', '', '172.23.88.131', 2882), is_user_sql=false) [2024-04-22 16:46:24.279974] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [13747][Occam][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584277018), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:46:09.241985, ballot number:27', 'HAPPENED_TIME', '2024-04-22 16:46:09.264468', '', '', '172.23.88.131', 2882)") [2024-04-22 16:46:24.279989] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [13747][Occam][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f4d7c0e8050, start=1713775584277060, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584277018), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:46:09.241985, ballot number:27', 'HAPPENED_TIME', '2024-04-22 16:46:09.264468', '', '', '172.23.88.131', 2882)) [2024-04-22 16:46:24.280059] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [13747][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584277018), 'ELECTION', 'lease expired', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'owner:"172.23.88.130:2882", expired time:16:46:09.241985, ballot number:27', 'HAPPENED_TIME', '2024-04-22 16:46:09.264468', '', '', '172.23.88.131', 2882), ret=-5019) [2024-04-22 16:46:24.281951] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=28][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:24.282020] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=52] table not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, table_name.ptr()="data_size:26, data:5F5F616C6C5F7365727665725F6576656E745F686973746F7279", ret=-5019) [2024-04-22 16:46:24.282073] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=49][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server_event_history, ret=-5019) [2024-04-22 16:46:24.282096] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=21][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server_event_history, db_name=oceanbase) [2024-04-22 16:46:24.282114] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server_event_history) [2024-04-22 16:46:24.282133] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:24.282145] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:24.282160] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_server_event_history' doesn't exist [2024-04-22 16:46:24.282172] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15090) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:24.282184] WDIAG [SQL.RESV] resolve_insert_field (ob_insert_resolver.cpp:438) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=11][errcode=-5019] fail to exec resolve_basic_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:24.282195] WDIAG [SQL.RESV] resolve_insert_clause (ob_insert_resolver.cpp:164) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=10][errcode=-5019] failed to resolve insert filed(ret=-5019) [2024-04-22 16:46:24.282205] WDIAG [SQL.RESV] resolve (ob_insert_resolver.cpp:96) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=9][errcode=-5019] resolve single table insert failed(ret=-5019) [2024-04-22 16:46:24.282217] WDIAG [SQL.RESV] stmt_resolver_func (ob_resolver.cpp:172) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3294) [2024-04-22 16:46:24.282236] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:24.282258] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=19][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:24.282272] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:24.282284] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:24.282296] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584280165), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:46:09.265075', '', '', '172.23.88.131', 2882), ret=-5019) [2024-04-22 16:46:24.282310] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:24.282321] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584280165), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:46:09.265075', '', '', '172.23.88.131', 2882)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.282346] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=20][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:24.282370] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:24.282381] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:24.282390] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:24.282428] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13747][Occam][T1][YB42AC175883-000616AB08F733EE-0-0] [lt=19][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584280165), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:46:09.265075', '', '', '172.23.88.131', 2882)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:24.282456] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13747][Occam][T0][YB42AC175883-000616AB08F733EE-0-0] [lt=22][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584280165), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:46:09.265075', '', '', '172.23.88.131', 2882)"}, aret=-5019, ret=-5019) [2024-04-22 16:46:24.282484] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1464) [13747][Occam][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584280165), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:46:09.265075', '', '', '172.23.88.131', 2882)) [2024-04-22 16:46:24.282529] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13747][Occam][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:24.282548] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1419) [13747][Occam][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584280165), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:46:09.265075', '', '', '172.23.88.131', 2882), is_user_sql=false) [2024-04-22 16:46:24.282571] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1408) [13747][Occam][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-5019] execute_write failed(ret=-5019, tenant_id=1, sql="INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584280165), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:46:09.265075', '', '', '172.23.88.131', 2882)") [2024-04-22 16:46:24.282592] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [13747][Occam][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-5019] execute sql failed(ret=-5019, conn=0x7f4d7c05c050, start=1713775584280197, sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584280165), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:46:09.265075', '', '', '172.23.88.131', 2882)) [2024-04-22 16:46:24.282726] WDIAG [SHARE] sync_add_event (ob_event_history_table_operator.h:311) [13747][Occam][T0][Y0-0000000000000000-0-0] [lt=85][errcode=-5019] sync execute sql failed(sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, value5, value6, svr_ip, svr_port) VALUES (usec_to_time(1713775584280165), 'ELECTION', 'vote', 'TENANT_ID', 1, 'LS_ID', 1, 'INFO', 'vote for "172.23.88.129:2882", reason:IP-PORT(priority equal)', 'HAPPENED_TIME', '2024-04-22 16:46:09.265075', '', '', '172.23.88.131', 2882), ret=-5019) [2024-04-22 16:46:24.300219] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.300309] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=0] will sleep(sleep_us=56000, remain_us=229861, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=56, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:24.312525] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:202) [14049][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=70] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f4d98ddc2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-04-22 16:46:24.346796] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.346903] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=17335026, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=170, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:24.360127] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.360213] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=57000, remain_us=169957, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=57, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:24.361256] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=68] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=5) [2024-04-22 16:46:24.362699] INFO [LIB] stat (utility.h:1165) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT WRITE LOG COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=4795, this=0x7f4d7e49f330, extra_info="palf_id:1") [2024-04-22 16:46:24.362761] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=59] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.131:2882", has_set_deleted:false}, accum_size=976) [2024-04-22 16:46:24.362821] INFO [LIB] stat (utility.h:1182) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=73, this=0x7f4d95343700, extra_info="") [2024-04-22 16:46:24.362951] INFO [LIB] stat (utility.h:1165) [13927][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=50] [PALF STAT FLUSH CB COST TIME](cur_stat_count=8, stat_interval=1000000, avg (count/cost)=77, this=0x7f4d7e49f560, extra_info="palf_id:1") [2024-04-22 16:46:24.363727] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175882-000616AB08DC3F77-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=24, this=0x7f4d7e4956a0, extra_info="palf_id:1") [2024-04-22 16:46:24.363793] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175882-000616AB08DC3F77-0-0] [lt=63] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=6201, this=0x7f4d7e4958d0, extra_info="palf_id:1") [2024-04-22 16:46:24.363774] INFO [PALF] read (log_cache.cpp:100) [14084][T1_ReplaySrv5][T1][Y0-0000000000000000-0-0] [lt=22] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2024-04-22 16:46:24.363823] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [17893][T1_L0_G1][T1][YB42AC175882-000616AB08DC3F77-0-0] [lt=27] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.131:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=20, avg_log_submit_to_flush_time=5030, avg_log_submit_to_slide_time=6181) [2024-04-22 16:46:24.363950] INFO [LIB] stat (utility.h:1165) [14080][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=80] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=84, this=0x7f4d98c048f8, extra_info="") [2024-04-22 16:46:24.364012] INFO [LIB] stat (utility.h:1165) [14080][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=60] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=10, this=0x7f4d98c04b28, extra_info="") [2024-04-22 16:46:24.406816] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=80] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:24.421361] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.421454] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1] will sleep(sleep_us=58000, remain_us=108716, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=58, v.err_=-4725, timeout_timestamp=1713775584530169) [2024-04-22 16:46:24.426406] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:24.426497] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=85][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775584156320380, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775584426388) [2024-04-22 16:46:24.426561] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=62][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775584226300, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:24.450612] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.450723] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=17231206, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=171, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:24.483826] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=1][errcode=-4012] [2024-04-22 16:46:24.483890] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=60][errcode=-4012] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4012, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:24.483950] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=57][errcode=-4012] failed to setup first scanner(ret=-4012) [2024-04-22 16:46:24.483976] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=23][errcode=-4012] Open this operator failed(ret=-4012, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:24.483995] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4012] open operator failed(ret=-4012) [2024-04-22 16:46:24.484007] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4012] fail open main query(ret=-4012) [2024-04-22 16:46:24.484018] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=11][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:46:24.484030] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:46:24.484041] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:24.484061] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=59, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.484085] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=21][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:59, local_retry_times:59, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-04-22 16:46:24.484119] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=24][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:46:24.484130] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=10][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:46:24.484140] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=9][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-04-22 16:46:24.484187] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330E-0-0] [lt=22][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-04-22 16:46:24.484216] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=23] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1952409) [2024-04-22 16:46:24.484240] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=23][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-04-22 16:46:24.484275] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=20][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:46:24.484291] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=15][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-04-22 16:46:24.484304] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:24.484319] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=12][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-04-22 16:46:24.484343] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=18][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:46:24.484442] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=16][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-04-22 16:46:24.484475] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=30][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:46:24.484509] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=31][errcode=-4012] get failed(ret=-4012) [2024-04-22 16:46:24.484526] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:888) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:46:24.484545] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4507) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=17][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:46:24.484560] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:521) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-04-22 16:46:24.484577] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:236) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330E-0-0] [lt=14][errcode=-4012] fail to process refresh task(ret=-4012, ret="OB_TIMEOUT", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-04-22 16:46:24.484602] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:599) [13685][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4012] fail to batch process task(ret=-4012) [2024-04-22 16:46:24.484614] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:499) [13685][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4012] fail to batch execute task(ret=-4012, tasks.count()=1) [2024-04-22 16:46:24.484733] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:261) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=11] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:46:24.484758] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:305) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=21] [REFRESH_SCHEMA] begin refresh schema, (begin_time=1713775584484757) [2024-04-22 16:46:24.484767] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2388) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=8] [REFRESH_SCHEMA] start to refresh and add schema(tenant_ids=[]) [2024-04-22 16:46:24.489860] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:24.489906] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.489932] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] already timeout, do not need sleep(sleep_us=0, remain_us=1994853, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.494438] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.494467] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=1000, remain_us=1990319, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.500034] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.500071] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=2000, remain_us=1984715, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.507383] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.507420] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=3000, remain_us=1977365, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.515079] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.515127] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=4000, remain_us=1969658, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.519450] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [14039][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=53] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-04-22 16:46:24.523922] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.523986] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=5000, remain_us=1960800, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.534448] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.534536] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=6000, remain_us=1950250, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.544471] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.544558] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=7000, remain_us=1940229, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.547349] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [13778][TsMgr][T0][Y0-0000000000000000-0-0] [lt=67] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:46:24.551538] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=47][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:24.551670] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=72] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:24.551745] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=56] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:46:24.551797] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=46][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:46:24.551838] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=38][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:46:24.551880] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=39][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:46:24.551912] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=23] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:24.551933] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=19][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:24.551953] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=18][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:24.551981] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=16][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:46:24.552000] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=16][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:24.552017] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=16][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:24.552035] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=17][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:24.552046] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:24.552058] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:24.552084] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=23][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:24.552096] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:24.552124] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=14][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:24.552145] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=17][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:24.552170] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=18][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:24.552191] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=18][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:24.552209] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=16][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-04-22 16:46:24.552243] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:24.552272] WDIAG [SERVER] do_process (obmp_query.cpp:760) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=28][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-04-22 16:46:24.552295] WDIAG [SERVER] do_process (obmp_query.cpp:884) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=17][errcode=-5019] query failed(ret=-5019, session={this:0x7f4d7edd01f8, id:3221225477, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-04-22 16:46:24.552342] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [14108][T1_L0_G0][T1][YB42AC175883-000616AB0B27322A-0-0] [lt=30] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0x579c808 0x578fbfb 0x5788b5f 0x577a048 0xabd78a4 0x16c398be 0x7f4dc8ea7ea5 0x7f4dc8bd0b0d") [2024-04-22 16:46:24.555346] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.555348] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [14021][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=114] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:46:24.555432] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=17126498, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=172, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:24.556516] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.556610] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=8000, remain_us=1928176, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.561398] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=53] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=4) [2024-04-22 16:46:24.568596] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.568725] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=9000, remain_us=1916061, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.580923] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.581014] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=10000, remain_us=1903773, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.586988] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [14060][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=23] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-04-22 16:46:24.587073] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [14060][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=58] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-04-22 16:46:24.587104] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:240) [14060][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=26] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=381) [2024-04-22 16:46:24.594762] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.594854] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=11000, remain_us=1889933, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.607440] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=79] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:24.610042] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.610154] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=12000, remain_us=1874632, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.626476] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.626573] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:24.626568] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=13000, remain_us=1858219, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.626613] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775584454700640, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775584626564) [2024-04-22 16:46:24.626639] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775584426583, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:24.626663] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:46:24.626652] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:310) [14064][T1_ArcFetcher][T1][YB42AC175883-000616AB07973224-0-0] [lt=33] ObArchiveFetcher is running(thread_index=0) [2024-04-22 16:46:24.629962] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [13930][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=105] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):65, sequence:2}) [2024-04-22 16:46:24.640236] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [13622][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=69][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:420, tid:13685}]) [2024-04-22 16:46:24.643869] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1][errcode=-4725] [2024-04-22 16:46:24.643915] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=43][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:24.643948] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:24.643967] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:24.643981] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:24.643993] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:24.644004] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.644049] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.644089] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=33][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.644125] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=33][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:24.644146] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=14, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.644180] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19] will sleep(sleep_us=14000, remain_us=1840607, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.646351] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:189) [14116][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=69][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-04-22 16:46:24.646462] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:445) [14116][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=74] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:46:24.646492] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [14116][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=30] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-04-22 16:46:24.655570] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=31][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:172, local_retry_times:172, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:24.655658] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=48][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.655680] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.655712] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=30][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:24.655740] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=15][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:24.658339] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:14, local_retry_times:14, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:24.658424] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.658462] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=33][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.658482] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:24.658543] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:24.659660] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] [2024-04-22 16:46:24.659689] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=27][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:24.659710] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:24.659726] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:24.659739] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:24.659747] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:24.659755] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=8][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.659789] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=7] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.659818] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=24][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.659827] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=8][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:24.659855] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=26][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=173, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.659881] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=14] will sleep(sleep_us=100000, remain_us=17022048, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=173, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:24.661854] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=47][errcode=-4725] [2024-04-22 16:46:24.661894] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=37][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:24.661954] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=55][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:24.661978] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:24.661998] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:24.662015] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:24.662033] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.662094] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.662164] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=60][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.662202] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=25][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:24.662233] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=29][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=15, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.662288] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=34] will sleep(sleep_us=15000, remain_us=1822500, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.677536] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=31][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:15, local_retry_times:15, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:24.677736] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=173][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.677768] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=30][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.677787] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:24.677820] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:24.681553] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=34][errcode=-4725] [2024-04-22 16:46:24.681599] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=43][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:24.681711] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=108][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:24.681735] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:24.681756] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:24.681818] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=59][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:24.681838] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.681901] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.682000] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=89][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.682033] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=28][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:24.682064] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=28][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=16, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.682107] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=26] will sleep(sleep_us=16000, remain_us=1802680, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.698364] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=29][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:16, local_retry_times:16, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:24.698454] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.698489] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=32][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.698519] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=26][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:24.698584] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:24.702727] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=29][errcode=-4725] [2024-04-22 16:46:24.702774] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=44][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:24.702835] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=57][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:24.702858] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:24.702879] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:24.702897] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:24.702914] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.702981] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.703051] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=60][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.703075] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:24.703102] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=24][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=17, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.703148] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=28] will sleep(sleep_us=17000, remain_us=1781639, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.720311] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:17, local_retry_times:17, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:24.720417] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=84][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.720454] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=33][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.720481] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=24][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:24.720534] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=22][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:24.724245] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=27][errcode=-4725] [2024-04-22 16:46:24.724285] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=37][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:24.724326] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=36][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:24.724350] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:24.724370] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:24.724387] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:24.724404] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.724466] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.724535] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=60][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.724555] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=17][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:24.724581] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=24][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=18, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.724664] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=28] will sleep(sleep_us=18000, remain_us=1760123, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.726498] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:249) [14034][T1_RFLWorker][T1][YB42AC175883-000616AB07373224-0-0] [lt=60] ObRemoteFetchWorker is running(thread_index=0) [2024-04-22 16:46:24.726697] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:24.726770] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=68][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775584454700640, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775584726682) [2024-04-22 16:46:24.726803] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775584626657, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:24.737679] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC2F-0-0] [lt=290][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:24.737759] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC2F-0-0] [lt=76][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:24.737837] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC2F-0-0] [lt=71][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:24.737965] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC2F-0-0] [lt=124][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:24.742821] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=42][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:18, local_retry_times:18, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:24.742901] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=60][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.742922] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.742934] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:24.742959] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:24.746446] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] [2024-04-22 16:46:24.746515] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=66][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:24.746547] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:24.746565] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:24.746579] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:24.746590] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:24.746602] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.746693] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.746745] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=44][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.746767] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:24.746787] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=19, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.746820] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19] will sleep(sleep_us=19000, remain_us=1737966, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.757671] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2271) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7AC-0-0] [lt=36][errcode=-4389] log_task life cost too much time(palf_id=1, self="172.23.88.131:2882", log_id=11293, log_task={header:{begin_lsn:{lsn:1376341}, end_lsn:{lsn:1376463}, log_id:11293, min_scn:{val:1713775584556335808, v:0}, max_scn:{val:1713775584556335808, v:0}, data_len:66, proposal_id:18, prev_lsn:{lsn:1376219}, prev_proposal_id:18, committed_end_lsn:{lsn:1376341}, data_checksum:667309872, accum_checksum:2154815907, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1713775584557153, freeze_ts:1713775584557153, submit_ts:1713775584557165, flushed_ts:1713775584561501, gen_to_freeze cost time:0, gen_to_submit cost time:12, submit_to_flush cost time:4336}, fs_cb_begin_ts=1713775584757584, log_life_time=200431) [2024-04-22 16:46:24.760012] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=14][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:173, local_retry_times:173, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:24.760087] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=56][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.760106] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=17][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.760116] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:24.760158] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:24.761594] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=67] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=3) [2024-04-22 16:46:24.763833] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=31][errcode=-4725] [2024-04-22 16:46:24.763879] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=43][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:24.763928] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=46][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:24.763953] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:24.763969] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:24.763987] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:24.764004] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.764066] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.764140] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=66][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.764163] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:24.764191] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=26][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=174, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.764235] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=28] will sleep(sleep_us=100000, remain_us=16917694, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=174, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:24.765946] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=34][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:19, local_retry_times:19, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:24.766067] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=100][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.766094] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=24][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.766108] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:24.766163] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:24.770116] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=26][errcode=-4725] [2024-04-22 16:46:24.770166] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=46][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:24.770205] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=36][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:24.770223] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:24.770236] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:24.770248] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:24.770259] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.770326] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=29] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.770374] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=40][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.770389] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:24.770407] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=20, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.770439] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18] will sleep(sleep_us=20000, remain_us=1714347, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.790594] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=38][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:20, local_retry_times:20, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:24.790730] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=114][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.790753] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.790766] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:24.790791] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:24.794550] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=56][errcode=-4725] [2024-04-22 16:46:24.794593] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:24.794621] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:24.794674] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=49][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:24.794688] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:24.794700] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:24.794711] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.794757] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.794797] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=34][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.794810] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:24.794828] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=21, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.794861] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18] will sleep(sleep_us=21000, remain_us=1689925, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.808184] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=66] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20814036992, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:24.816017] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=39][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:21, local_retry_times:21, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:24.816108] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=69][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.816131] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.816142] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:24.816190] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:24.820187] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=38][errcode=-4725] [2024-04-22 16:46:24.820240] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=51][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:24.820273] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:24.820298] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:24.820315] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:24.820368] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=50][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:24.820402] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.820452] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.820495] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=36][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.820510] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:24.820528] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=22, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.820563] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19] will sleep(sleep_us=22000, remain_us=1664223, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.824268] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=79] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=16000, clean_start_pos=1132461, clean_num=125829) [2024-04-22 16:46:24.842730] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:22, local_retry_times:22, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:24.842818] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.842846] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.842859] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:24.842885] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:24.845467] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=46][errcode=-4725] [2024-04-22 16:46:24.845508] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=39][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:24.845535] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=24][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:24.845551] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:24.845565] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:24.845580] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:24.845593] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.845699] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.845745] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=38][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.845761] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:24.845781] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=23, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.845828] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=33] will sleep(sleep_us=23000, remain_us=1638958, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.864408] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=34][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:174, local_retry_times:174, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:24.864536] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=102][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.864575] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=35][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.864597] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=20][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:24.864674] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=17][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:24.868506] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=31][errcode=-4725] [2024-04-22 16:46:24.868548] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=39][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:24.868578] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:24.868596] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:24.868609] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:24.868621] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:24.868688] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=64][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.868755] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.868854] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=89][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:24.868878] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=21][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:24.868904] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=23][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=175, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.868947] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=27] will sleep(sleep_us=100000, remain_us=16812982, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=175, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:24.868964] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:23, local_retry_times:23, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:24.869031] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=49][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.869052] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:24.869063] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:24.869086] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:24.872827] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] [2024-04-22 16:46:24.872878] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=49][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:24.872905] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:24.872921] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:24.872935] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:24.872986] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.873082] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=24000, remain_us=1611705, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.901671] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.901747] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=25000, remain_us=1583039, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.926926] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:24.927007] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=76][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775584750012158, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775584926909) [2024-04-22 16:46:24.927042] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775584726823, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:24.927139] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1713775584750012158, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1713775584927068}, version_delta=177126) [2024-04-22 16:46:24.930980] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.931073] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=26000, remain_us=1553714, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.935911] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=31][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:24.936018] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=82] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:46:24.936051] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-04-22 16:46:24.936279] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=219][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-04-22 16:46:24.936367] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=84][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-04-22 16:46:24.936407] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=23][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:24.936422] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=14][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:24.936441] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=11][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-04-22 16:46:24.936456] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:24.936468] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:24.936480] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=11][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:24.936492] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:4307) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=10][errcode=-5019] resolve table failed(ret=-5019) [2024-04-22 16:46:24.936504] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:3867) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=10][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-04-22 16:46:24.936516] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3723) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=10][errcode=-5019] resolve joined table failed(ret=-5019) [2024-04-22 16:46:24.936535] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=17][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:24.936554] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=17][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:24.936601] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=42][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:24.936621] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=17][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:24.936707] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=66][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:24.936724] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=12][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:24.936744] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=14][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:24.936759] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:24.936772] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-04-22 16:46:24.936850] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=73][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:24.936873] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=20][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:24.936910] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=29][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:24.936978] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=58][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:24.936999] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:24.937012] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=13][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:24.937050] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349E-0-0] [lt=12][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:24.937079] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [14114][BlackListServic][T0][YB42AC175883-000616AB07E7349E-0-0] [lt=24][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-04-22 16:46:24.937102] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:24.937133] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=0] start to print blacklist info [2024-04-22 16:46:24.937268] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:193) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=24] ls blacklist refresh finish(cost_time=3390) [2024-04-22 16:46:24.938231] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [14062][T1_LSArchiveMgr][T1][YB42AC175883-000616AB07773224-0-0] [lt=48] gc stale ls task succ [2024-04-22 16:46:24.940000] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [14156][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=14] current monitor number(seq_=-1) [2024-04-22 16:46:24.940017] INFO do_work (ob_rl_mgr.cpp:704) [13683][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=37] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:46:24.941428] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [14055][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=68] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=13240, mem_used=68631552) [2024-04-22 16:46:24.942292] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [14072][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=55] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:46:24.958310] INFO pn_ratelimit (group.c:589) [13817][IngressService][T0][Y0-0000000000000000-0-0] [lt=55] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-04-22 16:46:24.960547] WDIAG load_file_to_string (utility.h:627) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:46:24.960674] WDIAG get_ethernet_speed (utility.cpp:580) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=67][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:46:24.960717] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:46:24.960741] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:46:24.961411] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.961505] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=27000, remain_us=1523281, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:24.961858] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=135] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=2) [2024-04-22 16:46:24.972521] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.972578] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=16709351, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=176, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:24.992465] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:24.992568] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=28000, remain_us=1492218, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.024833] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.024950] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=29000, remain_us=1459836, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.024930] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20814036992, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:25.046964] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13787][BatchIO][T0][Y0-0000000000000000-0-0] [lt=64] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.047201] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13797][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=24] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.047292] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13802][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.047347] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13801][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=60] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.047500] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13794][BatchIO][T0][Y0-0000000000000000-0-0] [lt=57] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.047655] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13791][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.047671] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13793][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.047930] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13799][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=33] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.048087] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13806][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=50] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.048091] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13780][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.048106] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13796][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=40] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.048102] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13798][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.048104] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13783][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.048229] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13779][RpcIO][T0][Y0-0000000000000000-0-0] [lt=51] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.048677] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13800][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=27] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.048744] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13790][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.048876] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13785][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.049030] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13788][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.049063] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13792][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.049278] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13782][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.049457] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13804][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.049939] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13784][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.049945] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13781][RpcIO][T0][Y0-0000000000000000-0-0] [lt=49] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.050245] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13789][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:25.057688] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7B0-0-0] [lt=94] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x7f4d7e495470, extra_info="") [2024-04-22 16:46:25.057778] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7B0-0-0] [lt=59] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.131:2882", role="FOLLOWER", total_group_log_cnt=7, avg_log_batch_cnt=0, total_group_log_size=854, avg_group_log_size=122) [2024-04-22 16:46:25.057832] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7B0-0-0] [lt=49] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.131:2882", committed size=854) [2024-04-22 16:46:25.057913] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.058042] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=30000, remain_us=1426744, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.061974] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=6][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:25.062049] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=54] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:46:25.062098] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=44][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:46:25.062136] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=35][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:46:25.062175] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=36][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:46:25.062199] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=17][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:25.062211] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:25.062251] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=32][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:46:25.062264] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:25.062276] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:25.062287] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:25.062298] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:25.062310] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:25.062321] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:25.062333] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:25.062357] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:25.062385] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=24][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:25.062418] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13620][ConfigMgr][T1][YB42AC175883-000616AB09973979-0-0] [lt=28][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:25.072726] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=17][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:25.076470] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.076554] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=16605375, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=177, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:25.092159] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.092258] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=31000, remain_us=1392529, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.126868] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=56] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:46:25.127060] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:25.127128] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775584945119591, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775585127049) [2024-04-22 16:46:25.127166] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775584927066, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:25.127168] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.127202] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=32000, remain_us=1357584, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=32, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.140221] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [14059][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=16] ====== check clog disk timer task ====== [2024-04-22 16:46:25.140279] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [14059][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=48] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=65) [2024-04-22 16:46:25.140323] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [14059][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=22] cannot_recycle_log_size statistics(cannot_recycle_log_size=1376829, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:46:25.141760] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [13622][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=61][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:25, tid:13620}]) [2024-04-22 16:46:25.149860] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [13728][AutoLSLoc][T0][YB42AC175883-000616AB09E73395-0-0] [lt=378] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:18, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713775573499769, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"]) [2024-04-22 16:46:25.160785] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=13][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:25.160863] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=57] table not exist(tenant_id=1, database_id=201001, table_name=__all_disk_io_calibration, table_name.ptr()="data_size:25, data:5F5F616C6C5F6469736B5F696F5F63616C6962726174696F6E", ret=-5019) [2024-04-22 16:46:25.160896] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_disk_io_calibration, ret=-5019) [2024-04-22 16:46:25.160916] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_disk_io_calibration, db_name=oceanbase) [2024-04-22 16:46:25.160936] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_disk_io_calibration) [2024-04-22 16:46:25.160955] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:25.160967] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:25.160984] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_disk_io_calibration' doesn't exist [2024-04-22 16:46:25.161032] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=45][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:25.161047] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=14][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:25.161057] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=9][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:25.161069] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:25.161081] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:25.161093] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:25.161104] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:25.161131] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=14][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:25.161144] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:25.161158] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:25.161170] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:25.161182] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.131" and svr_port = 2882 and storage_name = "DATA", ret=-5019) [2024-04-22 16:46:25.161207] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=23][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:25.161220] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=12][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.131" and svr_port = 2882 and storage_name = "DATA""}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:25.161249] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=23][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:25.161286] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=31][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:25.161298] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:25.161308] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:25.161331] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13614][ServerGTimer][T1][YB42AC175883-000616AB0807345F-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.131" and svr_port = 2882 and storage_name = "DATA""}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:25.161359] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13614][ServerGTimer][T0][YB42AC175883-000616AB0807345F-0-0] [lt=24][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.131" and svr_port = 2882 and storage_name = "DATA""}, aret=-5019, ret=-5019) [2024-04-22 16:46:25.161383] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.131" and svr_port = 2882 and storage_name = "DATA") [2024-04-22 16:46:25.161405] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:25.161424] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:25.161460] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7f58a050, start=1713775585160501, sql=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.131" and svr_port = 2882 and storage_name = "DATA") [2024-04-22 16:46:25.161489] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:25.161504] WDIAG [COMMON] parse_calibration_table (ob_io_calibration.cpp:838) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, sql_string=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "172.23.88.131" and svr_port = 2882 and storage_name = "DATA") [2024-04-22 16:46:25.161702] WDIAG [COMMON] read_from_table (ob_io_calibration.cpp:708) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] parse calibration data failed(ret=-5019) [2024-04-22 16:46:25.161756] WDIAG [SERVER] refresh_io_calibration (ob_server.cpp:3426) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-5019] fail to refresh io calibration from table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:25.161781] WDIAG [SERVER] runTimerTask (ob_server.cpp:3416) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-5019] ObRefreshIOCalibrationTimeTask task failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:25.162009] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=47] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=1) [2024-04-22 16:46:25.163056] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.163120] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=33000, remain_us=1321667, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=33, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.180012] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.180048] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=16501881, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=178, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:25.182421] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [13693][LSMetaTblUp0][T0][YB42AC175883-000616AB08B9F80E-0-0] [lt=69][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713775570633812}) [2024-04-22 16:46:25.188588] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=41][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:25.188716] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=109] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2024-04-22 16:46:25.188756] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=35][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_unit, ret=-5019) [2024-04-22 16:46:25.188778] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=20][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_unit, db_name=oceanbase) [2024-04-22 16:46:25.188797] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_unit) [2024-04-22 16:46:25.188815] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:25.188828] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:25.188845] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_unit' doesn't exist [2024-04-22 16:46:25.188857] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:25.188868] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:25.188879] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:25.188890] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:25.188913] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=21][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:25.188925] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:25.188936] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:25.188961] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:25.188983] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=19][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:25.188999] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:25.189011] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:25.189022] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-04-22 16:46:25.189036] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:25.189057] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=19][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:25.189084] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:25.189109] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=20][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:25.189134] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=23][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:25.189144] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:25.189176] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:25.189204] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=23][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, aret=-5019, ret=-5019) [2024-04-22 16:46:25.189219] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-04-22 16:46:25.189232] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=12][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:25.189244] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:25.189257] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=11][errcode=-5019] query failed(ret=-5019, conn=0x7f4da0e3e050, start=1713775585188385, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-04-22 16:46:25.189274] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:25.189287] WDIAG [SHARE] read_units (ob_unit_table_operator.cpp:1177) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=9][errcode=-5019] execute sql failed(sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-04-22 16:46:25.189383] WDIAG [SHARE] get_units_by_tenant (ob_unit_table_operator.cpp:872) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=20][errcode=-5019] read_units failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-04-22 16:46:25.189400] WDIAG [SHARE] get_sys_unit_count (ob_unit_table_operator.cpp:68) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=15][errcode=-5019] failed to get units by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:25.189413] WDIAG [SHARE] get_sys_unit_count (ob_unit_getter.cpp:436) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=11][errcode=-5019] ut_operator get sys unit count failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:25.189425] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:94) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=11][errcode=-5019] get sys unit count fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:25.189438] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:108) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=11] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:25.191057] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=16][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:25.191119] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=47] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-04-22 16:46:25.191153] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=31][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, ret=-5019) [2024-04-22 16:46:25.191175] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=20][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_tenant, db_name=oceanbase) [2024-04-22 16:46:25.191193] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_tenant) [2024-04-22 16:46:25.191211] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:25.191241] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=29][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:25.191258] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_tenant' doesn't exist [2024-04-22 16:46:25.191269] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:25.191280] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:25.191291] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:25.191301] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:25.191312] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:25.191323] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:25.191334] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:25.191354] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:25.191366] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:25.191379] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:25.191391] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:25.191410] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=17][errcode=-5019] fail to handle text query(stmt=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-04-22 16:46:25.191425] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:25.191436] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:25.191466] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:25.191492] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:25.191503] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:25.191512] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:25.191533] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13887][OmtNodeBalancer][T1][YB42AC175883-000616AB06E732E3-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:25.191551] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, aret=-5019, ret=-5019) [2024-04-22 16:46:25.191565] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT tenant_id FROM __all_tenant) [2024-04-22 16:46:25.191578] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:25.191597] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=18][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:25.191611] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7c0fc050, start=1713775585190941, sql=SELECT tenant_id FROM __all_tenant) [2024-04-22 16:46:25.191661] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=48][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:25.191677] WDIAG [SHARE] read_tenants (ob_unit_table_operator.cpp:1240) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=13][errcode=-5019] execute sql failed(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-04-22 16:46:25.191755] WDIAG [SHARE] get_tenants (ob_unit_table_operator.cpp:109) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=13][errcode=-5019] fail read tenants(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-04-22 16:46:25.191775] WDIAG [SHARE] get_tenants (ob_unit_getter.cpp:198) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=12][errcode=-5019] ut_operator get_resource_pools failed(ret=-5019) [2024-04-22 16:46:25.191787] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:124) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=11][errcode=-5019] get cluster tenants fail(ret=-5019) [2024-04-22 16:46:25.191901] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:301) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=11] cal_all_part_disk_default_percentage succ(data_dir="/home/creality/oceanbase/store/sstable", clog_dir="/home/creality/oceanbase/store/clog", shared_mode=false, data_disk_total_size=205110317056, data_disk_default_percentage=90, clog_disk_total_size=85856358400, clog_disk_default_percentage=90) [2024-04-22 16:46:25.191946] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:337) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=42] decide disk size finished(suggested_disk_size=81604378624, suggested_disk_percentage=0, default_disk_percentage=90, total_space=85856358400, disk_size=81604378624) [2024-04-22 16:46:25.191972] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=24] get_log_disk_info_in_config(suggested_data_disk_size=161061273600, suggested_clog_disk_size=81604378624, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=81604378624, log_disk_percentage=0, total_log_disk_size=85856358400) [2024-04-22 16:46:25.191995] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=19] try_resize success(ret=0, log_disk_size=81604378624, total_log_disk_size=85856358400, this={dir::"/home/creality/oceanbase/store/clog/log_pool", dir_fd:14, meta_fd:15, log_pool_meta:{curr_total_size:81604378624, next_total_size:81604378624, status:0}, min_block_id:2, max_block_id:1216, min_log_disk_size_for_all_tenants_:4294967296, is_inited:true}) [2024-04-22 16:46:25.192033] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:132) [13887][OmtNodeBalancer][T0][YB42AC175883-000616AB06E732E3-0-0] [lt=36] refresh tenant config(tenants=[], ret=-5019) [2024-04-22 16:46:25.193009] INFO eloop_run (eloop.c:140) [13667][pnio1][T0][Y0-0000000000000000-0-0] [lt=73] PNIO [ratelimit] time: 1713775585193005, bytes: 91634972, bw: 0.026186 MB/s, add_ts: 1007628, add_bytes: 27667 [2024-04-22 16:46:25.192809] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2375) [13886][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=748] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"2GB", log_disk_size:"4GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3}}, mode:0, create_timestamp:1713773681580721, is_removed:false}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=9, offset=1469}, ls_meta_entry:[5](ver=0,mode=0,seq=4), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:false, version:3, snapshot_cnt:0}, create_status:1}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:35, min_worker_cnt:14, max_worker_cnt:150, stopped:0, worker_us:1027908670, recv_hp_rpc_cnt:17, recv_np_rpc_cnt:96, recv_lp_rpc_cnt:0, recv_mysql_cnt:829, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:2192983, workers:14, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:2, queue_size:0, recv_req_cnt:3740, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773684738461}{group_id:10, queue_size:0, recv_req_cnt:5260, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773689876615}{group_id:5, queue_size:0, recv_req_cnt:46, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773750875165}{group_id:100, queue_size:0, recv_req_cnt:7730, min_worker_cnt:8, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1713774763918974}{group_id:9, queue_size:0, recv_req_cnt:27450, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1713773676271171}{group_id:1, queue_size:0, recv_req_cnt:20745, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1713775545956063}, rpc_stat_info: pcode=0x1501:cnt=50 pcode=0x1516:cnt=47 pcode=0x4a9:cnt=30 pcode=0x14a:cnt=24 pcode=0x150a:cnt=20, token_change_ts:1713773676235309, tenant_role:1}) [2024-04-22 16:46:25.193915] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2375) [13886][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=1102] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:5, max_cpu:5, memory_size:"1GB", log_disk_size:"2GB", min_iops:10000, max_iops:50000, iops_weight:0}}, mode:0, create_timestamp:1713773672333063, is_removed:false}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:true, version:3, snapshot_cnt:0}, create_status:1}, unit_min_cpu:"5.000000000000000000e+00", unit_max_cpu:"5.000000000000000000e+00", total_worker_cnt:30, min_worker_cnt:22, max_worker_cnt:150, stopped:0, worker_us:249219120, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3669049, workers:22, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1713773672345113, tenant_role:0}) [2024-04-22 16:46:25.194887] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2375) [13886][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=968] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:2.5, max_cpu:2.5, memory_size:"1GB", log_disk_size:"2GB", min_iops:10000, max_iops:50000, iops_weight:0}}, mode:0, create_timestamp:1713773672312930, is_removed:false}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:true, version:3, snapshot_cnt:0}, create_status:1}, unit_min_cpu:"2.500000000000000000e+00", unit_max_cpu:"2.500000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:167267741, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:1834153, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1713773672323914, tenant_role:0}) [2024-04-22 16:46:25.199036] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.199129] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=34000, remain_us=1285658, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=34, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.201481] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:32) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=9] === Run print tenant memory usage task === [2024-04-22 16:46:25.201586] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=41] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 581,132,288 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 188,243,968 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 6,291,456 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 115,343,360 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 8,388,608 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 14,680,064 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 33,783,808 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 11,292,672 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 201,011,200 limit= 9,223,372,036,854,775,807 [2024-04-22 16:46:25.202514] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=99] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 188,243,968 used= 174,379,344 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31,477,760 used= 31,458,304 count= 1 avg_used= 31,458,304 block_cnt= 1 chunk_cnt= 1 mod=ash_list [MEMORY] hold= 22,761,472 used= 22,699,200 count= 9 avg_used= 2,522,133 block_cnt= 9 chunk_cnt= 8 mod=IoControl [MEMORY] hold= 15,704,064 used= 14,669,736 count= 213 avg_used= 68,872 block_cnt= 213 chunk_cnt= 28 mod=[T]ObSessionDIB [MEMORY] hold= 12,873,728 used= 12,856,352 count= 1 avg_used= 12,856,352 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 9,457,664 used= 9,437,784 count= 1 avg_used= 9,437,784 block_cnt= 1 chunk_cnt= 1 mod=HashBuckInteChe [MEMORY] hold= 5,299,136 used= 5,219,688 count= 47 avg_used= 111,057 block_cnt= 24 chunk_cnt= 10 mod=PartitTableTask [MEMORY] hold= 5,267,456 used= 5,190,496 count= 8 avg_used= 648,812 block_cnt= 8 chunk_cnt= 3 mod=KvstCachWashStr [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashBucTenComMo [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashPxBlooFilKe [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=PxP2PDhMgrKey [MEMORY] hold= 4,307,328 used= 4,278,711 count= 3 avg_used= 1,426,237 block_cnt= 3 chunk_cnt= 3 mod=SqlDtlMgr [MEMORY] hold= 4,231,936 used= 4,203,008 count= 6 avg_used= 700,501 block_cnt= 5 chunk_cnt= 3 mod=BaseLogWriter [MEMORY] hold= 4,214,880 used= 4,194,328 count= 2 avg_used= 2,097,164 block_cnt= 2 chunk_cnt= 2 mod=SerFuncRegHT [MEMORY] hold= 3,276,800 used= 3,214,848 count= 11 avg_used= 292,258 block_cnt= 11 chunk_cnt= 7 mod=DedupQueue [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBucIdUnitMa [MEMORY] hold= 2,167,040 used= 2,129,912 count= 7 avg_used= 304,273 block_cnt= 7 chunk_cnt= 5 mod=FixedQueue [MEMORY] hold= 2,114,048 used= 2,031,616 count= 1,026 avg_used= 1,980 block_cnt= 208 chunk_cnt= 3 mod=TabletLSMap [MEMORY] hold= 2,037,344 used= 1,915,008 count= 468 avg_used= 4,091 block_cnt= 243 chunk_cnt= 3 mod=CharsetInit [MEMORY] hold= 1,622,016 used= 1,441,792 count= 22 avg_used= 65,536 block_cnt= 22 chunk_cnt= 10 mod=[T]char [MEMORY] hold= 1,591,232 used= 1,560,960 count= 21 avg_used= 74,331 block_cnt= 21 chunk_cnt= 9 mod=TenantCtxAlloca [MEMORY] hold= 1,581,056 used= 1,572,904 count= 1 avg_used= 1,572,904 block_cnt= 1 chunk_cnt= 1 mod=IdConnMap [MEMORY] hold= 1,523,712 used= 1,507,118 count= 5 avg_used= 301,423 block_cnt= 5 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 1,452,608 used= 1,434,880 count= 44 avg_used= 32,610 block_cnt= 44 chunk_cnt= 18 mod=CommonArray [MEMORY] hold= 1,408,800 used= 1,388,000 count= 4 avg_used= 347,000 block_cnt= 4 chunk_cnt= 4 mod=Omt [MEMORY] hold= 1,286,144 used= 1,270,296 count= 2 avg_used= 635,148 block_cnt= 2 chunk_cnt= 2 mod=PxResMgr [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=TenantResCtrl [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=ConcurHashMap [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=TsSourceInfoMap [MEMORY] hold= 1,056,768 used= 1,048,608 count= 1 avg_used= 1,048,608 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 974,976 used= 944,384 count= 476 avg_used= 1,984 block_cnt= 124 chunk_cnt= 23 mod=CreateContext [MEMORY] hold= 933,888 used= 917,576 count= 2 avg_used= 458,788 block_cnt= 2 chunk_cnt= 1 mod=CACHE_INST_MAP [MEMORY] hold= 827,472 used= 773,160 count= 13 avg_used= 59,473 block_cnt= 12 chunk_cnt= 7 mod=HashBucket [MEMORY] hold= 761,856 used= 733,184 count= 4 avg_used= 183,296 block_cnt= 4 chunk_cnt= 4 mod=LightyQueue [MEMORY] hold= 704,512 used= 691,200 count= 2 avg_used= 345,600 block_cnt= 2 chunk_cnt= 2 mod=LDBlockBitMap [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=GEleTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=EventTimer [MEMORY] hold= 622,592 used= 619,008 count= 1 avg_used= 619,008 block_cnt= 1 chunk_cnt= 1 mod=SysTaskStatus [MEMORY] hold= 512,000 used= 507,904 count= 64 avg_used= 7,936 block_cnt= 64 chunk_cnt= 2 mod=ModulePageAlloc [MEMORY] hold= 491,264 used= 441,808 count= 271 avg_used= 1,630 block_cnt= 118 chunk_cnt= 15 mod=tg [MEMORY] hold= 458,752 used= 457,856 count= 7 avg_used= 65,408 block_cnt= 7 chunk_cnt= 4 mod=ObTsResponseTas [MEMORY] hold= 442,368 used= 235,224 count= 27 avg_used= 8,712 block_cnt= 27 chunk_cnt= 10 mod=[T]ObDSActionAr [MEMORY] hold= 425,984 used= 420,032 count= 1 avg_used= 420,032 block_cnt= 1 chunk_cnt= 1 mod=CompEventMgr [MEMORY] hold= 409,600 used= 371,040 count= 10 avg_used= 37,104 block_cnt= 10 chunk_cnt= 6 mod=InneSqlConnPool [MEMORY] hold= 403,648 used= 378,880 count= 258 avg_used= 1,468 block_cnt= 54 chunk_cnt= 2 mod=LSLocationMap [MEMORY] hold= 401,408 used= 393,256 count= 1 avg_used= 393,256 block_cnt= 1 chunk_cnt= 1 mod=TablStorStatMgr [MEMORY] hold= 368,896 used= 330,944 count= 7 avg_used= 47,277 block_cnt= 7 chunk_cnt= 3 mod=Rpc [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=register_task [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=register_tasks [MEMORY] hold= 229,376 used= 224,000 count= 1 avg_used= 224,000 block_cnt= 1 chunk_cnt= 1 mod=BGTMonitor [MEMORY] hold= 221,184 used= 212,992 count= 1 avg_used= 212,992 block_cnt= 1 chunk_cnt= 1 mod=TSWorker [MEMORY] hold= 221,184 used= 214,432 count= 1 avg_used= 214,432 block_cnt= 1 chunk_cnt= 1 mod=CompSuggestMgr [MEMORY] hold= 212,992 used= 196,624 count= 2 avg_used= 98,312 block_cnt= 2 chunk_cnt= 2 mod=DdlQue [MEMORY] hold= 212,992 used= 204,928 count= 1 avg_used= 204,928 block_cnt= 1 chunk_cnt= 1 mod=TenantMutilAllo [MEMORY] hold= 212,992 used= 196,624 count= 2 avg_used= 98,312 block_cnt= 2 chunk_cnt= 1 mod=DRTaskMap [MEMORY] hold= 203,872 used= 202,200 count= 26 avg_used= 7,776 block_cnt= 26 chunk_cnt= 7 mod=HashNode [MEMORY] hold= 196,608 used= 195,576 count= 3 avg_used= 65,192 block_cnt= 3 chunk_cnt= 3 mod=LatchStat [MEMORY] hold= 180,224 used= 172,064 count= 1 avg_used= 172,064 block_cnt= 1 chunk_cnt= 1 mod=TenantMBList [MEMORY] hold= 163,840 used= 147,792 count= 2 avg_used= 73,896 block_cnt= 2 chunk_cnt= 1 mod=HashNodNexWaiMa [MEMORY] hold= 163,504 used= 155,416 count= 2 avg_used= 77,708 block_cnt= 2 chunk_cnt= 2 mod=OB_DISK_REP [MEMORY] hold= 160,624 used= 159,272 count= 21 avg_used= 7,584 block_cnt= 21 chunk_cnt= 2 mod=TenaSpaTabIdSet [MEMORY] hold= 155,648 used= 147,624 count= 1 avg_used= 147,624 block_cnt= 1 chunk_cnt= 1 mod=UsrRuleMap [MEMORY] hold= 155,648 used= 153,152 count= 3 avg_used= 51,050 block_cnt= 3 chunk_cnt= 2 mod=IO_MGR [MEMORY] hold= 144,880 used= 143,656 count= 19 avg_used= 7,560 block_cnt= 19 chunk_cnt= 2 mod=SysTableNameMap [MEMORY] hold= 139,264 used= 131,776 count= 1 avg_used= 131,776 block_cnt= 1 chunk_cnt= 1 mod=GtsTaskQueue [MEMORY] hold= 137,920 used= 134,912 count= 11 avg_used= 12,264 block_cnt= 11 chunk_cnt= 4 mod=SeArray [MEMORY] hold= 122,880 used= 120,408 count= 3 avg_used= 40,136 block_cnt= 3 chunk_cnt= 3 mod=CommSysVarFac [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=TenaSchForCacMa [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=MemMgrForLiboMa [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=MemMgrMap [MEMORY] hold= 122,560 used= 118,904 count= 2 avg_used= 59,452 block_cnt= 2 chunk_cnt= 2 mod=RefrFullScheMap [MEMORY] hold= 115,616 used= 114,656 count= 15 avg_used= 7,643 block_cnt= 15 chunk_cnt= 3 mod=HashNodeConfCon [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=IndNameMap [MEMORY] hold= 114,688 used= 111,096 count= 1 avg_used= 111,096 block_cnt= 1 chunk_cnt= 1 mod=NonPartTenMap [MEMORY] hold= 114,336 used= 106,088 count= 2 avg_used= 53,044 block_cnt= 2 chunk_cnt= 2 mod=RetryCtrl [MEMORY] hold= 106,496 used= 98,312 count= 1 avg_used= 98,312 block_cnt= 1 chunk_cnt= 1 mod=TmpFileManager [MEMORY] hold= 106,496 used= 86,408 count= 5 avg_used= 17,281 block_cnt= 5 chunk_cnt= 2 mod=HashBuckConfCon [MEMORY] hold= 96,768 used= 95,424 count= 21 avg_used= 4,544 block_cnt= 21 chunk_cnt= 9 mod=[T]ObTraceEvent [MEMORY] hold= 96,000 used= 95,232 count= 12 avg_used= 7,936 block_cnt= 12 chunk_cnt= 6 mod=RsEventQueue [MEMORY] hold= 88,000 used= 56,704 count= 10 avg_used= 5,670 block_cnt= 9 chunk_cnt= 6 mod=BucketLock [MEMORY] hold= 81,920 used= 74,064 count= 2 avg_used= 37,032 block_cnt= 2 chunk_cnt= 2 mod=io_trace_map [MEMORY] hold= 73,728 used= 69,664 count= 1 avg_used= 69,664 block_cnt= 1 chunk_cnt= 1 mod=SuperBlockBuffe [MEMORY] hold= 73,728 used= 65,600 count= 1 avg_used= 65,600 block_cnt= 1 chunk_cnt= 1 mod=TCREF [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=ObLSLocation [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=KVCACHE_HAZARD [MEMORY] hold= 65,136 used= 63,096 count= 2 avg_used= 31,548 block_cnt= 2 chunk_cnt= 2 mod=ScheCacSysCacMa [MEMORY] hold= 63,808 used= 55,440 count= 6 avg_used= 9,240 block_cnt= 6 chunk_cnt= 3 mod=MallocInfoMap [MEMORY] hold= 57,344 used= 55,368 count= 1 avg_used= 55,368 block_cnt= 1 chunk_cnt= 1 mod=DepInfoTaskQ [MEMORY] hold= 55,968 used= 27,136 count= 377 avg_used= 71 block_cnt= 216 chunk_cnt= 21 mod=Coro [MEMORY] hold= 49,152 used= 36,912 count= 2 avg_used= 18,456 block_cnt= 2 chunk_cnt= 2 mod=HashBuckSysConf [MEMORY] hold= 49,152 used= 37,032 count= 3 avg_used= 12,344 block_cnt= 3 chunk_cnt= 2 mod=ReferedMap [MEMORY] hold= 49,152 used= 32,768 count= 2 avg_used= 16,384 block_cnt= 2 chunk_cnt= 1 mod=CACHE_TNT_LST [MEMORY] hold= 48,000 used= 47,616 count= 6 avg_used= 7,936 block_cnt= 6 chunk_cnt= 4 mod=SessionInfoHash [MEMORY] hold= 46,912 used= 37,632 count= 18 avg_used= 2,090 block_cnt= 14 chunk_cnt= 5 mod=TGTimer [MEMORY] hold= 45,760 used= 45,056 count= 11 avg_used= 4,096 block_cnt= 11 chunk_cnt= 5 mod=LinearHashMapDi [MEMORY] hold= 45,760 used= 45,056 count= 11 avg_used= 4,096 block_cnt= 11 chunk_cnt= 6 mod=LinearHashMapCn [MEMORY] hold= 41,264 used= 37,264 count= 2 avg_used= 18,632 block_cnt= 2 chunk_cnt= 1 mod=TaskRunnerSer [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=ObLongopsMgr [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DDLSpeedCtrl [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=TmpFileStoreMap [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucPooUniMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=ClientSessBuck [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=KillSessMapBuck [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SessHoldMapBuck [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SqlLoadData [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HasBucSerMigUnM [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucTenPooMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucSerUniMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucNamPooMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucIdPoolMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucConPooMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HasBucConRefCoM [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucNamConMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HashBucIdConfMa [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=T3MTBMap [MEMORY] hold= 40,000 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 3 mod=SqlSession [MEMORY] hold= 26,672 used= 25,280 count= 21 avg_used= 1,203 block_cnt= 13 chunk_cnt= 9 mod=ObGuard [MEMORY] hold= 24,944 used= 20,784 count= 2 avg_used= 10,392 block_cnt= 2 chunk_cnt= 2 mod=SqlNio [MEMORY] hold= 24,672 used= 24,264 count= 6 avg_used= 4,044 block_cnt= 6 chunk_cnt= 2 mod=ScheObSchemAren [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=FuncRuleMap [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=GrpNameIdMap [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=leakMap [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=ServerCkptSlogH [MEMORY] hold= 24,576 used= 18,456 count= 1 avg_used= 18,456 block_cnt= 1 chunk_cnt= 1 mod=GrpIdNameMap [MEMORY] hold= 24,224 used= 21,672 count= 2 avg_used= 10,836 block_cnt= 2 chunk_cnt= 1 mod=SchemaStatuMap [MEMORY] hold= 22,400 used= 5,088 count= 212 avg_used= 24 block_cnt= 130 chunk_cnt= 22 mod=[T]MemoryContex [MEMORY] hold= 21,504 used= 20,352 count= 16 avg_used= 1,272 block_cnt= 16 chunk_cnt= 2 mod=IO_GROUP_MAP [MEMORY] hold= 18,416 used= 18,144 count= 4 avg_used= 4,536 block_cnt= 4 chunk_cnt= 3 mod=DeviceMng [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=HasBucTimZonInM [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=GenSchemVersMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TbltRefreshMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MemDumpMap [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=LinkArray [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 9,336 count= 1 avg_used= 9,336 block_cnt= 1 chunk_cnt= 1 mod=InnerLobHash [MEMORY] hold= 16,384 used= 10,592 count= 1 avg_used= 10,592 block_cnt= 1 chunk_cnt= 1 mod=TenCompProgMgr [MEMORY] hold= 16,384 used= 8,992 count= 1 avg_used= 8,992 block_cnt= 1 chunk_cnt= 1 mod=IO_HEALTH [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=ServerLogPool [MEMORY] hold= 16,384 used= 12,296 count= 1 avg_used= 12,296 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 16,032 used= 15,904 count= 2 avg_used= 7,952 block_cnt= 2 chunk_cnt= 2 mod=UpgProcSet [MEMORY] hold= 16,000 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 1 mod=CommSysVarDefVa [MEMORY] hold= 16,000 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 1 mod=MacroIdList [MEMORY] hold= 16,000 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 2 mod=PlanVaIdx [MEMORY] hold= 11,744 used= 8,944 count= 43 avg_used= 208 block_cnt= 28 chunk_cnt= 5 mod=Scheduler [MEMORY] hold= 10,816 used= 9,984 count= 13 avg_used= 768 block_cnt= 13 chunk_cnt= 4 mod=timer [MEMORY] hold= 9,472 used= 8,384 count= 16 avg_used= 524 block_cnt= 4 chunk_cnt= 2 mod=RpcBuffer [MEMORY] hold= 8,384 used= 8,104 count= 4 avg_used= 2,026 block_cnt= 3 chunk_cnt= 3 mod=TenantInfo [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerRegioMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=SqlSessiVarMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=Node [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=SchemaRowKey [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=AddrList [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerCidMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=RetryTask [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerBlacklist [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PrepareOk [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ServerIdcMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=acceptOK [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ObTsTenantInfoN [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=followerRenew [MEMORY] hold= 7,968 used= 7,904 count= 1 avg_used= 7,904 block_cnt= 1 chunk_cnt= 1 mod=LuaHandler [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HasNodTzInfM [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=SessHoldMapNode [MEMORY] hold= 7,840 used= 7,776 count= 1 avg_used= 7,776 block_cnt= 1 chunk_cnt= 1 mod=ClientSessNode [MEMORY] hold= 6,704 used= 3,840 count= 40 avg_used= 96 block_cnt= 29 chunk_cnt= 9 mod=PThread [MEMORY] hold= 5,424 used= 5,288 count= 2 avg_used= 2,644 block_cnt= 2 chunk_cnt= 2 mod=DeadLock [MEMORY] hold= 5,376 used= 5,248 count= 2 avg_used= 2,624 block_cnt= 1 chunk_cnt= 1 mod=RootContext [MEMORY] hold= 5,152 used= 4,704 count= 7 avg_used= 672 block_cnt= 6 chunk_cnt= 6 mod=[T]ObWarningBuf [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmReq [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=SqlPx [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=RebuildCtx [MEMORY] hold= 3,888 used= 3,816 count= 1 avg_used= 3,816 block_cnt= 1 chunk_cnt= 1 mod=RecScheHisMap [MEMORY] hold= 3,888 used= 3,816 count= 1 avg_used= 3,816 block_cnt= 1 chunk_cnt= 1 mod=RemMasterMap [MEMORY] hold= 3,744 used= 2,808 count= 13 avg_used= 216 block_cnt= 8 chunk_cnt= 6 mod=ObFuture [MEMORY] hold= 3,264 used= 3,200 count= 1 avg_used= 3,200 block_cnt= 1 chunk_cnt= 1 mod=TenantTZ [MEMORY] hold= 2,768 used= 2,704 count= 1 avg_used= 2,704 block_cnt= 1 chunk_cnt= 1 mod=LoggerAlloc [MEMORY] hold= 2,400 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=StorageS3 [MEMORY] hold= 1,728 used= 1,536 count= 3 avg_used= 512 block_cnt= 3 chunk_cnt= 2 mod=TntResourceMgr [MEMORY] hold= 1,600 used= 1,536 count= 1 avg_used= 1,536 block_cnt= 1 chunk_cnt= 1 mod=LobManager [MEMORY] hold= 1,504 used= 1,432 count= 1 avg_used= 1,432 block_cnt= 1 chunk_cnt= 1 mod=GtsRequestRpc [MEMORY] hold= 1,472 used= 1,408 count= 1 avg_used= 1,408 block_cnt= 1 chunk_cnt= 1 mod=TsMgr [MEMORY] hold= 1,472 used= 1,400 count= 1 avg_used= 1,400 block_cnt= 1 chunk_cnt= 1 mod=GtiRequestRpc [MEMORY] hold= 1,472 used= 1,408 count= 1 avg_used= 1,408 block_cnt= 1 chunk_cnt= 1 mod=T3MQSyncLock [MEMORY] hold= 1,408 used= 1,344 count= 1 avg_used= 1,344 block_cnt= 1 chunk_cnt= 1 mod=SchemaService [MEMORY] hold= 1,392 used= 1,328 count= 1 avg_used= 1,328 block_cnt= 1 chunk_cnt= 1 mod=GtiRpcProxy [MEMORY] hold= 1,392 used= 1,328 count= 1 avg_used= 1,328 block_cnt= 1 chunk_cnt= 1 mod=GtsRpcProxy [MEMORY] hold= 1,360 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=TENANT_PLAN_MAP [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=INGRESS_MAP [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=IO_CHANNEL_MAP [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GROUP_INDEX_MAP [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 1,184 used= 1,088 count= 1 avg_used= 1,088 block_cnt= 1 chunk_cnt= 1 mod=memdumpqueue [MEMORY] hold= 1,168 used= 1,092 count= 1 avg_used= 1,092 block_cnt= 1 chunk_cnt= 1 mod=LogRegionMap [MEMORY] hold= 768 used= 704 count= 1 avg_used= 704 block_cnt= 1 chunk_cnt= 1 mod=ScheMgrCacheMap [MEMORY] hold= 640 used= 280 count= 5 avg_used= 56 block_cnt= 5 chunk_cnt= 1 mod=Log [MEMORY] hold= 576 used= 512 count= 1 avg_used= 512 block_cnt= 1 chunk_cnt= 1 mod=SqlString [MEMORY] hold= 576 used= 512 count= 1 avg_used= 512 block_cnt= 1 chunk_cnt= 1 mod=SqlSessiQuerSql [MEMORY] hold= 528 used= 264 count= 3 avg_used= 88 block_cnt= 3 chunk_cnt= 2 mod=DestKAState [MEMORY] hold= 528 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=ContextsMap [MEMORY] hold= 496 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=SequenceIdMap [MEMORY] hold= 496 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=PackStateMap [MEMORY] hold= 496 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=SequenceMap [MEMORY] hold= 416 used= 272 count= 2 avg_used= 136 block_cnt= 1 chunk_cnt= 1 mod=unknown [MEMORY] hold= 352 used= 192 count= 2 avg_used= 96 block_cnt= 2 chunk_cnt= 1 mod=ObFunction [MEMORY] hold= 288 used= 135 count= 2 avg_used= 67 block_cnt= 1 chunk_cnt= 1 mod=EventReHolder [MEMORY] hold= 288 used= 84 count= 3 avg_used= 28 block_cnt= 2 chunk_cnt= 2 mod=KeepAliveServer [MEMORY] hold= 240 used= 112 count= 2 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=UserResourceMgr [MEMORY] hold= 240 used= 48 count= 3 avg_used= 16 block_cnt= 3 chunk_cnt= 3 mod=CreateEntity [MEMORY] hold= 224 used= 160 count= 1 avg_used= 160 block_cnt= 1 chunk_cnt= 1 mod=OccamTimeGuard [MEMORY] hold= 128 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=PxTargetMgr [MEMORY] hold= 80 used= 7 count= 1 avg_used= 7 block_cnt= 1 chunk_cnt= 1 mod=SqlExpr [MEMORY] hold= 80 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=Compressor [MEMORY] hold= 174,379,344 used= 171,180,395 count= 4,210 avg_used= 40,660 mod=SUMMARY [2024-04-22 16:46:25.202972] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=341] [MEMORY] tenant_id= 500 ctx_id= GLIBC hold= 6,291,456 used= 2,553,888 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 1,600,016 used= 675,052 count= 12,703 avg_used= 53 block_cnt= 209 chunk_cnt= 2 mod=Buffer [MEMORY] hold= 900,352 used= 640,650 count= 2,744 avg_used= 233 block_cnt= 188 chunk_cnt= 3 mod=glibc_malloc [MEMORY] hold= 53,520 used= 37,729 count= 229 avg_used= 164 block_cnt= 29 chunk_cnt= 1 mod=S3SDK [MEMORY] hold= 2,553,888 used= 1,353,431 count= 15,676 avg_used= 86 mod=SUMMARY [2024-04-22 16:46:25.203029] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=30] [MEMORY] tenant_id= 500 ctx_id= CO_STACK hold= 115,343,360 used= 109,412,352 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 109,412,352 used= 109,208,832 count= 212 avg_used= 515,136 block_cnt= 212 chunk_cnt= 55 mod=CoStack [MEMORY] hold= 109,412,352 used= 109,208,832 count= 212 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:46:25.203081] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=25] [MEMORY] tenant_id= 500 ctx_id= LIBEASY hold= 8,388,608 used= 3,882,960 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 3,882,960 used= 3,662,624 count= 265 avg_used= 13,821 block_cnt= 48 chunk_cnt= 4 mod=OB_TEST2_PCODE [MEMORY] hold= 3,882,960 used= 3,662,624 count= 265 avg_used= 13,821 mod=SUMMARY [2024-04-22 16:46:25.203165] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=54] [MEMORY] tenant_id= 500 ctx_id= LOGGER_CTX_ID hold= 14,680,064 used= 14,565,376 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10,403,840 used= 10,398,720 count= 5 avg_used= 2,079,744 block_cnt= 5 chunk_cnt= 5 mod=Logger [MEMORY] hold= 4,161,536 used= 4,159,488 count= 2 avg_used= 2,079,744 block_cnt= 2 chunk_cnt= 2 mod=ErrorLogger [MEMORY] hold= 14,565,376 used= 14,558,208 count= 7 avg_used= 2,079,744 mod=SUMMARY [2024-04-22 16:46:25.203254] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=18] [MEMORY] tenant_id= 500 ctx_id= RPC_CTX_ID hold= 2,097,152 used= 16,384 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16,384 used= 16,256 count= 1 avg_used= 16,256 block_cnt= 1 chunk_cnt= 1 mod=RpcDefault [MEMORY] hold= 16,384 used= 16,256 count= 1 avg_used= 16,256 mod=SUMMARY [2024-04-22 16:46:25.203331] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=31] [MEMORY] tenant_id= 500 ctx_id= PKT_NIO hold= 33,783,808 used= 29,929,936 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 25,465,296 used= 25,315,936 count= 38 avg_used= 666,208 block_cnt= 22 chunk_cnt= 12 mod=DEFAULT [MEMORY] hold= 1,843,200 used= 1,833,664 count= 14 avg_used= 130,976 block_cnt= 14 chunk_cnt= 4 mod=PKTS_INBUF [MEMORY] hold= 1,835,008 used= 1,833,664 count= 14 avg_used= 130,976 block_cnt= 14 chunk_cnt= 4 mod=PKTC_INBUF [MEMORY] hold= 229,376 used= 228,032 count= 14 avg_used= 16,288 block_cnt= 14 chunk_cnt= 4 mod=SERVER_CTX_CHUN [MEMORY] hold= 229,376 used= 228,032 count= 14 avg_used= 16,288 block_cnt= 14 chunk_cnt= 4 mod=CLIENT_CB_CHUNK [MEMORY] hold= 196,608 used= 195,456 count= 12 avg_used= 16,288 block_cnt= 12 chunk_cnt= 2 mod=SERVER_RESP_CHU [MEMORY] hold= 131,072 used= 130,304 count= 8 avg_used= 16,288 block_cnt= 8 chunk_cnt= 2 mod=CLIENT_REQ_CHUN [MEMORY] hold= 29,929,936 used= 29,765,088 count= 114 avg_used= 261,097 mod=SUMMARY [2024-04-22 16:46:25.203677] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=21] [MEMORY] tenant_id= 500 ctx_id= SCHEMA_SERVICE hold= 11,292,672 used= 9,760,320 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 7,098,368 used= 7,078,824 count= 1 avg_used= 7,078,824 block_cnt= 1 chunk_cnt= 1 mod=SchemaIdVersion [MEMORY] hold= 2,088,768 used= 2,087,680 count= 2 avg_used= 1,043,840 block_cnt= 2 chunk_cnt= 2 mod=TenantSchemMgr [MEMORY] hold= 360,448 used= 327,680 count= 4 avg_used= 81,920 block_cnt= 4 chunk_cnt= 1 mod=SchemaMgrCache [MEMORY] hold= 110,208 used= 102,397 count= 5 avg_used= 20,479 block_cnt= 3 chunk_cnt= 1 mod=SchemaSysCache [MEMORY] hold= 31,872 used= 31,616 count= 4 avg_used= 7,904 block_cnt= 4 chunk_cnt= 1 mod=ScheTenaInfoVec [MEMORY] hold= 16,320 used= 16,064 count= 4 avg_used= 4,016 block_cnt= 4 chunk_cnt= 1 mod=SchemaSysVariab [MEMORY] hold= 15,936 used= 15,808 count= 2 avg_used= 7,904 block_cnt= 2 chunk_cnt= 1 mod=ScheTablInfoVec [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheIndeNameMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheTablIdMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=HiddenTblNames [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheRoutIdMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheRoutNameMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=ScheTablNameMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=SchePackIdMap [MEMORY] hold= 2,176 used= 2,048 count= 2 avg_used= 1,024 block_cnt= 2 chunk_cnt= 1 mod=SchePackNameMap [MEMORY] hold= 1,536 used= 1,024 count= 8 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabeSeCompo [MEMORY] hold= 1,152 used= 768 count= 6 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabeSePolic [MEMORY] hold= 1,152 used= 1,024 count= 2 avg_used= 512 block_cnt= 2 chunk_cnt= 1 mod=ScheUdtNameMap [MEMORY] hold= 1,152 used= 1,024 count= 2 avg_used= 512 block_cnt= 2 chunk_cnt= 1 mod=ScheUdtIdMap [MEMORY] hold= 1,152 used= 768 count= 6 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabeSeLabel [MEMORY] hold= 800 used= 512 count= 4 avg_used= 128 block_cnt= 3 chunk_cnt= 1 mod=SchemaSynonym [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=DBLINK_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheLabSeUserLe [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 4 chunk_cnt= 1 mod=DIRECTORY_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=RLS_POLICY_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheOutlSqlMap [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=RLS_GROUP_MGR [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaProfile [MEMORY] hold= 768 used= 512 count= 4 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=RLS_CONTEXT_MGR [MEMORY] hold= 656 used= 584 count= 1 avg_used= 584 block_cnt= 1 chunk_cnt= 1 mod=TenSchMemMgrFoL [MEMORY] hold= 656 used= 584 count= 1 avg_used= 584 block_cnt= 1 chunk_cnt= 1 mod=TenaScheMemMgr [MEMORY] hold= 416 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheDataNameMap [MEMORY] hold= 416 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=MockFkParentTab [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaSecurAudi [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheAuxVpNameVe [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheForKeyNamMa [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaContext [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheConsNameMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheOutlIdMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheOutlNameMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchePriTabPriMa [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchePriObjPriMa [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheTrigIdMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaTablespac [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=ScheTrigNameMap [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaUdf [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaSequence [MEMORY] hold= 384 used= 256 count= 2 avg_used= 128 block_cnt= 2 chunk_cnt= 1 mod=SchemaKeystore [MEMORY] hold= 9,760,320 used= 9,691,189 count= 134 avg_used= 72,322 mod=SUMMARY [2024-04-22 16:46:25.203863] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=122] [MEMORY] tenant_id= 500 ctx_id= UNEXPECTED_IN_500 hold= 201,011,200 used= 199,030,144 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 134,238,288 used= 134,217,736 count= 2 avg_used= 67,108,868 block_cnt= 2 chunk_cnt= 2 mod=CACHE_MAP_BKT [MEMORY] hold= 18,051,072 used= 18,033,840 count= 1 avg_used= 18,033,840 block_cnt= 1 chunk_cnt= 1 mod=CACHE_MB_HANDLE [MEMORY] hold= 17,329,216 used= 8,968,960 count= 1,093 avg_used= 8,205 block_cnt= 1,093 chunk_cnt= 9 mod=StorageLoggerM [MEMORY] hold= 16,806,784 used= 16,786,208 count= 3 avg_used= 5,595,402 block_cnt= 3 chunk_cnt= 2 mod=FixeSizeBlocAll [MEMORY] hold= 6,311,936 used= 6,291,472 count= 1 avg_used= 6,291,472 block_cnt= 1 chunk_cnt= 1 mod=CACHE_MAP_LOCK [MEMORY] hold= 3,699,088 used= 3,287,104 count= 86 avg_used= 38,222 block_cnt= 54 chunk_cnt= 4 mod=OccamThreadPool [MEMORY] hold= 2,543,616 used= 2,523,256 count= 1 avg_used= 2,523,256 block_cnt= 1 chunk_cnt= 1 mod=TenantConfig [MEMORY] hold= 36,160 used= 35,968 count= 2 avg_used= 17,984 block_cnt= 2 chunk_cnt= 2 mod=CommonNetwork [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=BlockMap [MEMORY] hold= 5,984 used= 1,088 count= 68 avg_used= 16 block_cnt= 2 chunk_cnt= 2 mod=ConfigChecker [MEMORY] hold= 199,030,144 used= 190,153,568 count= 1,258 avg_used= 151,155 mod=SUMMARY [2024-04-22 16:46:25.204002] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=75] [MEMORY] tenant: 508, limit: 1,073,741,824 hold: 23,650,304 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 6,873,088 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 16,777,216 limit= 9,223,372,036,854,775,807 [2024-04-22 16:46:25.204110] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=33] [MEMORY] tenant_id= 508 ctx_id= DEFAULT_CTX_ID hold= 6,873,088 used= 5,621,856 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,678,784 used= 2,658,688 count= 1 avg_used= 2,658,688 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,211,840 used= 2,066,160 count= 30 avg_used= 68,872 block_cnt= 30 chunk_cnt= 2 mod=[T]ObSessionDIB [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DynamicFactor [MEMORY] hold= 14,400 used= 12,480 count= 30 avg_used= 416 block_cnt= 4 chunk_cnt= 1 mod=OMT_Worker [MEMORY] hold= 8,160 used= 3,840 count= 60 avg_used= 64 block_cnt= 4 chunk_cnt= 1 mod=Coro [MEMORY] hold= 3,008 used= 720 count= 30 avg_used= 24 block_cnt= 4 chunk_cnt= 1 mod=[T]MemoryContex [MEMORY] hold= 1,152 used= 1,080 count= 1 avg_used= 1,080 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 5,621,856 used= 5,439,200 count= 154 avg_used= 35,319 mod=SUMMARY [2024-04-22 16:46:25.204273] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=52] [MEMORY] tenant_id= 508 ctx_id= CO_STACK hold= 16,777,216 used= 15,482,880 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 15,482,880 used= 15,454,080 count= 30 avg_used= 515,136 block_cnt= 30 chunk_cnt= 8 mod=CoStack [MEMORY] hold= 15,482,880 used= 15,454,080 count= 30 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:46:25.204548] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=67] [MEMORY] tenant: 509, limit: 1,073,741,824 hold: 17,358,848 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 6,873,088 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 10,485,760 limit= 9,223,372,036,854,775,807 [2024-04-22 16:46:25.204662] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=31] [MEMORY] tenant_id= 509 ctx_id= DEFAULT_CTX_ID hold= 6,873,088 used= 4,875,968 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2,678,784 used= 2,658,688 count= 1 avg_used= 2,658,688 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 1,474,560 used= 1,377,440 count= 20 avg_used= 68,872 block_cnt= 20 chunk_cnt= 2 mod=[T]ObSessionDIB [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DynamicFactor [MEMORY] hold= 9,600 used= 8,320 count= 20 avg_used= 416 block_cnt= 3 chunk_cnt= 1 mod=OMT_Worker [MEMORY] hold= 5,440 used= 2,560 count= 40 avg_used= 64 block_cnt= 3 chunk_cnt= 1 mod=Coro [MEMORY] hold= 1,920 used= 480 count= 20 avg_used= 24 block_cnt= 3 chunk_cnt= 1 mod=[T]MemoryContex [MEMORY] hold= 1,152 used= 1,080 count= 1 avg_used= 1,080 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 4,875,968 used= 4,744,800 count= 104 avg_used= 45,623 mod=SUMMARY [2024-04-22 16:46:25.204830] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=37] [MEMORY] tenant_id= 509 ctx_id= CO_STACK hold= 10,485,760 used= 10,321,920 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 10,321,920 used= 10,302,720 count= 20 avg_used= 515,136 block_cnt= 20 chunk_cnt= 5 mod=CoStack [MEMORY] hold= 10,321,920 used= 10,302,720 count= 20 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:46:25.205162] INFO [LIB] operator() (ob_malloc_allocator.cpp:537) [13619][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=30] [MEMORY] tenant: 1, limit: 4,294,967,296 hold: 588,111,872 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 411,930,624 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= EXECUTE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 111,149,056 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 2,097,152 limit= 858,993,440 [MEMORY] ctx_id= LOB_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 18,894,848 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33,554,432 limit= 9,223,372,036,854,775,807 [2024-04-22 16:46:25.206173] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=46] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 411,930,624 used= 361,493,360 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 20 [MEMORY] hold= 144,539,648 used= 144,483,008 count= 33 avg_used= 4,378,272 block_cnt= 33 chunk_cnt= 33 mod=MysqlRequesReco [MEMORY] hold= 41,963,520 used= 41,943,040 count= 1 avg_used= 41,943,040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 23,744,512 used= 23,680,320 count= 24 avg_used= 986,680 block_cnt= 24 chunk_cnt= 12 mod=IoControl [MEMORY] hold= 15,114,240 used= 14,118,760 count= 205 avg_used= 68,872 block_cnt= 205 chunk_cnt= 29 mod=[T]ObSessionDIB [MEMORY] hold= 14,483,248 used= 14,360,745 count= 93 avg_used= 154,416 block_cnt= 33 chunk_cnt= 7 mod=OmtTenant [MEMORY] hold= 8,986,624 used= 8,945,664 count= 2 avg_used= 4,472,832 block_cnt= 2 chunk_cnt= 2 mod=ServerObjecPool [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8,540,160 used= 8,519,680 count= 1 avg_used= 8,519,680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 8,257,536 used= 8,240,064 count= 2 avg_used= 4,120,032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 4,943,872 used= 4,915,456 count= 2 avg_used= 2,457,728 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 4,571,136 used= 4,525,056 count= 6 avg_used= 754,176 block_cnt= 6 chunk_cnt= 5 mod=ResourceGroup [MEMORY] hold= 4,235,264 used= 4,194,304 count= 2 avg_used= 2,097,152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 4,214,784 used= 4,194,336 count= 1 avg_used= 4,194,336 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 3,162,112 used= 3,145,808 count= 2 avg_used= 1,572,904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3,145,728 used= 2,228,224 count= 128 avg_used= 17,408 block_cnt= 128 chunk_cnt= 4 mod=SqlDtlQueue [MEMORY] hold= 2,678,784 used= 2,658,688 count= 1 avg_used= 2,658,688 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2,654,208 used= 2,359,296 count= 36 avg_used= 65,536 block_cnt= 36 chunk_cnt= 7 mod=[T]char [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2,375,680 used= 2,359,536 count= 2 avg_used= 1,179,768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2,248,704 used= 2,228,224 count= 1 avg_used= 2,228,224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2,198,864 used= 416,700 count= 25,277 avg_used= 16 block_cnt= 270 chunk_cnt= 2 mod=Number [MEMORY] hold= 2,166,784 used= 2,142,208 count= 2 avg_used= 1,071,104 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 2,080,768 used= 2,079,744 count= 1 avg_used= 2,079,744 block_cnt= 1 chunk_cnt= 1 mod=LogReplayTask [MEMORY] hold= 1,843,200 used= 1,770,696 count= 9 avg_used= 196,744 block_cnt= 9 chunk_cnt= 2 mod=DagMap [MEMORY] hold= 1,777,664 used= 1,770,696 count= 1 avg_used= 1,770,696 block_cnt= 1 chunk_cnt= 1 mod=TableProc [MEMORY] hold= 1,596,912 used= 1,588,632 count= 3 avg_used= 529,544 block_cnt= 3 chunk_cnt= 2 mod=TabletMap [MEMORY] hold= 1,584,288 used= 1,575,992 count= 3 avg_used= 525,330 block_cnt= 2 chunk_cnt= 1 mod=PartitTableTask [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1,294,336 used= 1,280,384 count= 2 avg_used= 640,192 block_cnt= 2 chunk_cnt= 2 mod=TransTimeWheel [MEMORY] hold= 1,269,184 used= 1,263,904 count= 6 avg_used= 210,650 block_cnt= 6 chunk_cnt= 2 mod=PoolFreeList [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 1,187,840 used= 1,179,768 count= 1 avg_used= 1,179,768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 958,464 used= 950,272 count= 1 avg_used= 950,272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 933,888 used= 931,072 count= 1 avg_used= 931,072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 802,816 used= 800,000 count= 1 avg_used= 800,000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 786,560 used= 524,352 count= 33 avg_used= 15,889 block_cnt= 33 chunk_cnt= 3 mod=LogAlloc [MEMORY] hold= 663,552 used= 659,200 count= 1 avg_used= 659,200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 663,552 used= 655,360 count= 1 avg_used= 655,360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 655,360 used= 650,752 count= 1 avg_used= 650,752 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647,168 used= 640,192 count= 1 avg_used= 640,192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 598,016 used= 590,232 count= 1 avg_used= 590,232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 442,368 used= 438,432 count= 3 avg_used= 146,144 block_cnt= 3 chunk_cnt= 2 mod=SQLSessionInfo [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409,600 used= 401,408 count= 1 avg_used= 401,408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 403,648 used= 378,880 count= 258 avg_used= 1,468 block_cnt= 53 chunk_cnt= 3 mod=LSMap [MEMORY] hold= 385,024 used= 375,520 count= 2 avg_used= 187,760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 385,024 used= 377,120 count= 1 avg_used= 377,120 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303,104 used= 294,936 count= 1 avg_used= 294,936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 212,992 used= 212,160 count= 1 avg_used= 212,160 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 210,912 used= 209,248 count= 26 avg_used= 8,048 block_cnt= 26 chunk_cnt= 9 mod=[T]ObPerfEventR [MEMORY] hold= 204,800 used= 196,744 count= 1 avg_used= 196,744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 204,800 used= 196,616 count= 1 avg_used= 196,616 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 196,608 used= 196,224 count= 3 avg_used= 65,408 block_cnt= 3 chunk_cnt= 2 mod=StorMetaCacheIO [MEMORY] hold= 179,712 used= 177,216 count= 39 avg_used= 4,544 block_cnt= 39 chunk_cnt= 11 mod=[T]ObTraceEvent [MEMORY] hold= 152,000 used= 150,784 count= 19 avg_used= 7,936 block_cnt= 19 chunk_cnt= 9 mod=LSIter [MEMORY] hold= 147,456 used= 139,264 count= 1 avg_used= 139,264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 147,456 used= 78,408 count= 9 avg_used= 8,712 block_cnt= 9 chunk_cnt= 6 mod=[T]ObDSActionAr [MEMORY] hold= 122,880 used= 110,608 count= 2 avg_used= 55,304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 114,688 used= 106,496 count= 1 avg_used= 106,496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 106,496 used= 100,008 count= 1 avg_used= 100,008 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 98,304 used= 96,032 count= 1 avg_used= 96,032 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 94,240 used= 90,112 count= 64 avg_used= 1,408 block_cnt= 17 chunk_cnt= 5 mod=TxCtxMgr [MEMORY] hold= 81,920 used= 65,536 count= 2 avg_used= 32,768 block_cnt= 2 chunk_cnt= 1 mod=CsDictPl [MEMORY] hold= 73,728 used= 72,736 count= 1 avg_used= 72,736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73,728 used= 66,176 count= 1 avg_used= 66,176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 73,728 used= 65,536 count= 1 avg_used= 65,536 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoMgr [MEMORY] hold= 72,240 used= 1,024 count= 896 avg_used= 1 block_cnt= 10 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=FlushMeta [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=FlushLog [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65,472 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=ResultSet [MEMORY] hold= 55,408 used= 26,512 count= 394 avg_used= 67 block_cnt= 103 chunk_cnt= 26 mod=Coro [MEMORY] hold= 49,152 used= 48,576 count= 1 avg_used= 48,576 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 48,832 used= 44,840 count= 2 avg_used= 22,420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=CsStrPl [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=RawPl [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=CsIntPl [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=HB_SERVICE [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=DictPl [MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40,960 used= 32,768 count= 1 avg_used= 32,768 block_cnt= 1 chunk_cnt= 1 mod=ConstPl [MEMORY] hold= 40,000 used= 39,680 count= 5 avg_used= 7,936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 32,768 used= 17,408 count= 2 avg_used= 8,704 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTTP [MEMORY] hold= 32,768 used= 25,600 count= 1 avg_used= 25,600 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 32,768 used= 24,688 count= 2 avg_used= 12,344 block_cnt= 2 chunk_cnt= 2 mod=TLD_ClientTask [MEMORY] hold= 32,576 used= 26,079 count= 2 avg_used= 13,039 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 28,224 used= 27,792 count= 6 avg_used= 4,632 block_cnt= 6 chunk_cnt= 2 mod=WorkerMap [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=IntDiffPl [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24,576 used= 16,384 count= 1 avg_used= 16,384 block_cnt= 1 chunk_cnt= 1 mod=RlePl [MEMORY] hold= 21,360 used= 19,488 count= 29 avg_used= 672 block_cnt= 12 chunk_cnt= 6 mod=[T]ObWarningBuf [MEMORY] hold= 20,080 used= 4,920 count= 205 avg_used= 24 block_cnt= 83 chunk_cnt= 24 mod=[T]MemoryContex [MEMORY] hold= 17,472 used= 16,128 count= 21 avg_used= 768 block_cnt= 18 chunk_cnt= 6 mod=TGTimer [MEMORY] hold= 16,800 used= 8,448 count= 3 avg_used= 2,816 block_cnt= 3 chunk_cnt= 3 mod=BaseLogWriter [MEMORY] hold= 16,800 used= 14,560 count= 35 avg_used= 416 block_cnt= 11 chunk_cnt= 4 mod=OMT_Worker [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=StrPrefixPl [MEMORY] hold= 16,384 used= 13,696 count= 1 avg_used= 13,696 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TLD_TableCtxMgr [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=HexStrPl [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=StrDiffPl [MEMORY] hold= 16,384 used= 11,584 count= 1 avg_used= 11,584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TabletStats [MEMORY] hold= 16,384 used= 12,288 count= 1 avg_used= 12,288 block_cnt= 1 chunk_cnt= 1 mod=DiskIteratorSto [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxIMgr [MEMORY] hold= 16,384 used= 8,448 count= 1 avg_used= 8,448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=ColEqualPl [MEMORY] hold= 16,384 used= 8,192 count= 1 avg_used= 8,192 block_cnt= 1 chunk_cnt= 1 mod=ColSubStrPl [MEMORY] hold= 16,384 used= 10,248 count= 1 avg_used= 10,248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16,384 used= 12,344 count= 1 avg_used= 12,344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 16,000 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 1 mod=SqlSession [MEMORY] hold= 15,744 used= 15,616 count= 2 avg_used= 7,808 block_cnt= 2 chunk_cnt= 1 mod=HashNodeLCSta [MEMORY] hold= 8,320 used= 8,192 count= 2 avg_used= 4,096 block_cnt= 2 chunk_cnt= 2 mod=LinearHashMapCn [MEMORY] hold= 8,320 used= 8,192 count= 2 avg_used= 4,096 block_cnt= 2 chunk_cnt= 2 mod=LinearHashMapDi [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=FetchLogTask [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PurgeThrottle [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=SqlSessiVarMap [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=PlanVaIdx [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=HandleSubmit [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MdsTableMgr [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoNode [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=HashNodePlanCac [MEMORY] hold= 7,872 used= 7,808 count= 1 avg_used= 7,808 block_cnt= 1 chunk_cnt= 1 mod=TabletIDSetNode [MEMORY] hold= 7,856 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletNode [MEMORY] hold= 7,856 used= 7,792 count= 1 avg_used= 7,792 block_cnt= 1 chunk_cnt= 1 mod=ColUsagNode [MEMORY] hold= 5,184 used= 5,120 count= 1 avg_used= 5,120 block_cnt= 1 chunk_cnt= 1 mod=CtxBlockPl [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4,704 used= 4,632 count= 1 avg_used= 4,632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4,160 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=CsCtxBlockPl [MEMORY] hold= 4,160 used= 4,096 count= 1 avg_used= 4,096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 3,072 used= 3,008 count= 1 avg_used= 3,008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2,400 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=CaLsInfoMap [MEMORY] hold= 2,400 used= 2,328 count= 1 avg_used= 2,328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2,336 used= 1,728 count= 8 avg_used= 216 block_cnt= 4 chunk_cnt= 3 mod=LogIOTask [MEMORY] hold= 1,984 used= 1,920 count= 1 avg_used= 1,920 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 1,872 used= 1,800 count= 1 avg_used= 1,800 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1,664 used= 1,600 count= 1 avg_used= 1,600 block_cnt= 1 chunk_cnt= 1 mod=LogPartFetCtxPo [MEMORY] hold= 1,616 used= 1,544 count= 1 avg_used= 1,544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1,488 used= 1,352 count= 2 avg_used= 676 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 1,472 used= 1,408 count= 1 avg_used= 1,408 block_cnt= 1 chunk_cnt= 1 mod=LockMemtableMgr [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GROUP_INDEX_MAP [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1,344 used= 1,272 count= 1 avg_used= 1,272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1,184 used= 1,056 count= 2 avg_used= 528 block_cnt= 1 chunk_cnt= 1 mod=PoolArenaArray [MEMORY] hold= 1,152 used= 1,080 count= 1 avg_used= 1,080 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1,120 used= 1,056 count= 1 avg_used= 1,056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1,040 used= 720 count= 5 avg_used= 144 block_cnt= 1 chunk_cnt= 1 mod=ObTxLSLogCb [MEMORY] hold= 848 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=TLD_TIMER [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=CheckPointTimer [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=FlushTimer [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=DiskUsageTimer [MEMORY] hold= 832 used= 768 count= 1 avg_used= 768 block_cnt= 1 chunk_cnt= 1 mod=TabShellTimer [MEMORY] hold= 720 used= 648 count= 1 avg_used= 648 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 624 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 576 used= 512 count= 1 avg_used= 512 block_cnt= 1 chunk_cnt= 1 mod=SqlSessiQuerSql [MEMORY] hold= 544 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 496 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=PackStateMap [MEMORY] hold= 496 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=SequenceMap [MEMORY] hold= 496 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=ContextsMap [MEMORY] hold= 496 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=SequenceIdMap [MEMORY] hold= 480 used= 120 count= 5 avg_used= 24 block_cnt= 3 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 448 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 448 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 448 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 336 used= 144 count= 2 avg_used= 72 block_cnt= 2 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 192 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 176 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 176 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 160 used= 16 count= 2 avg_used= 8 block_cnt= 2 chunk_cnt= 1 mod=ObLogEXTHandler [MEMORY] hold= 128 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 128 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 112 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 80 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 361,493,360 used= 355,691,661 count= 28,089 avg_used= 12,663 mod=SUMMARY [2024-04-22 16:46:25.206505] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=283] [MEMORY] tenant_id= 1 ctx_id= EXECUTE_CTX_ID hold= 2,097,152 used= 16,000 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 16,000 used= 15,872 count= 2 avg_used= 7,936 block_cnt= 2 chunk_cnt= 1 mod=SqlExecContext [MEMORY] hold= 16,000 used= 15,872 count= 2 avg_used= 7,936 mod=SUMMARY [2024-04-22 16:46:25.206540] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=18] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2,097,152 used= 794,624 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 794,624 used= 788,032 count= 1 avg_used= 788,032 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 794,624 used= 788,032 count= 1 avg_used= 788,032 mod=SUMMARY [2024-04-22 16:46:25.206591] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=22] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 2,097,152 used= 650,112 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 612,096 used= 553,704 count= 21 avg_used= 26,366 block_cnt= 21 chunk_cnt= 1 mod=SqlPhyPlan [MEMORY] hold= 24,000 used= 23,808 count= 3 avg_used= 7,936 block_cnt= 3 chunk_cnt= 1 mod=SqlPlanCache [MEMORY] hold= 12,288 used= 11,904 count= 6 avg_used= 1,984 block_cnt= 2 chunk_cnt= 1 mod=CreateContext [MEMORY] hold= 1,728 used= 1,600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 650,112 used= 591,016 count= 32 avg_used= 18,469 mod=SUMMARY [2024-04-22 16:46:25.206685] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=17] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2,097,152 used= 80,992 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80,896 used= 59,018 count= 263 avg_used= 224 block_cnt= 20 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80,992 used= 59,050 count= 264 avg_used= 223 mod=SUMMARY [2024-04-22 16:46:25.206740] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=27] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 111,149,056 used= 106,831,872 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 106,831,872 used= 106,633,152 count= 207 avg_used= 515,136 block_cnt= 207 chunk_cnt= 53 mod=CoStack [MEMORY] hold= 106,831,872 used= 106,633,152 count= 207 avg_used= 515,136 mod=SUMMARY [2024-04-22 16:46:25.206841] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=41] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 2,097,152 used= 466,944 limit= 858,993,440 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 401,408 used= 400,064 count= 2 avg_used= 200,032 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=N_TabletPool [MEMORY] hold= 466,944 used= 465,472 count= 3 avg_used= 155,157 mod=SUMMARY [2024-04-22 16:46:25.206966] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=51] [MEMORY] tenant_id= 1 ctx_id= LOB_CTX_ID hold= 2,097,152 used= 65,536 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 block_cnt= 1 chunk_cnt= 1 mod=LobAllocator [MEMORY] hold= 65,536 used= 65,408 count= 1 avg_used= 65,408 mod=SUMMARY [2024-04-22 16:46:25.207119] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=62] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 18,894,848 used= 17,870,464 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 17,862,464 used= 17,833,792 count= 3 avg_used= 5,944,597 block_cnt= 3 chunk_cnt= 2 mod=MEMTABLE_ARENA [MEMORY] hold= 8,000 used= 7,936 count= 1 avg_used= 7,936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 17,870,464 used= 17,841,728 count= 4 avg_used= 4,460,432 mod=SUMMARY [2024-04-22 16:46:25.207274] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:178) [13619][MemDumpTimer][T1][Y0-0000000000000000-0-0] [lt=38] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33,554,432 used= 0 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 33,554,432 free_size= 33,554,432 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2024-04-22 16:46:25.207460] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:122) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=32] ====== tenants memory info ====== === TENANTS MEMORY INFO === divisive_memory_used= 75,759,616 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 581,132,288 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 23,650,304 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 17,358,848 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,713,775,585,205,124 active_memstore_used= 0 total_memstore_used= 0 total_memstore_hold= 0 memstore_freeze_trigger_limit= 343,597,360 memstore_limit= 1,717,986,880 mem_tenant_limit= 4,294,967,296 mem_tenant_hold= 588,111,872 max_mem_memstore_can_get_now= 1,853,427,700 memstore_alloc_pos= 0 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2024-04-22 16:46:25.207993] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:162) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=68] [CHUNK_MGR] free=20 pushes=603670 pops=603650 limit= 26,843,545,600 hold= 1,256,689,664 total_hold= 1,348,468,736 used= 1,210,253,312 freelist_hold= 46,436,352 large_freelist_hold= 4,493,312 maps= 513 unmaps= 126 large_maps= 34 large_unmaps= 5 huge_maps= 32 huge_unmaps= 3 memalign=0 resident_size= 1,688,793,088 virtual_memory_used= 2,214,035,456 [2024-04-22 16:46:25.208097] INFO print (ob_malloc_time_monitor.cpp:39) [13619][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=61] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 0, 10): delta_total_cost_time= 2666008, delta_count= 19665962, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 10, 100): delta_total_cost_time= 21508, delta_count= 1498, avg_cost_time= 14 [MALLOC_TIME_MONITOR] [ 100, 1000): delta_total_cost_time= 117, delta_count= 1, avg_cost_time= 117 [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2024-04-22 16:46:25.209861] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=10] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:18, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713775573499769, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"]) [2024-04-22 16:46:25.212002] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=160][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:25.212073] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=53] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:46:25.212104] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=28][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:46:25.212132] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=26][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:46:25.212159] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=25][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:46:25.212186] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=20][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:25.212205] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=17][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:25.212229] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=15][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:46:25.212246] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=15][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:25.212264] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=17][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:25.212280] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=14][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:25.212320] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=38][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:25.212338] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=17][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:25.212356] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=16][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:25.212375] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=17][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:25.212409] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=18][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:25.212428] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=16][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:25.212451] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=19][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:25.212470] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=17][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:25.212487] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=14][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:46:25.212509] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=19][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:25.212528] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=16][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:25.212564] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=30][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:25.212621] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=48][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:25.212686] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=61][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:25.212702] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=15][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:25.212751] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=17][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:25.212778] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=25][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:46:25.212802] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=21][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:25.212824] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=20][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:25.212843] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=17][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:25.212863] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=18][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7c0e8050, start=1713775585211806, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:25.212888] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=23][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:25.212908] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=16][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:25.213048] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=39][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:46:25.213072] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=21][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:46:25.213095] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=21][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:25.213115] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=18][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:25.213138] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=18][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:46:25.213161] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=19][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:46:25.213180] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173931-0-0] [lt=18][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:25.225607] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=66] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20814036992, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:25.233445] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.233508] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=60] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.233537] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.233560] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.233582] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.233580] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC32-0-0] [lt=250][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:25.233695] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=110] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.233714] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.233727] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.233740] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.233753] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.233702] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC32-0-0] [lt=118][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:25.233773] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC32-0-0] [lt=67][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:25.233766] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.233817] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=48] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.233830] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.233810] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC32-0-0] [lt=35][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:25.233843] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.233856] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.233883] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:25.237789] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.237947] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=35000, remain_us=1246840, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=35, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.264514] INFO eloop_run (eloop.c:140) [13674][pnio2][T0][Y0-0000000000000000-0-0] [lt=72] PNIO [ratelimit] time: 1713775585264506, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007585, add_bytes: 0 [2024-04-22 16:46:25.277302] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.277389] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=36000, remain_us=1207397, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=36, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.277378] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [13720][SysLocAsyncUp0][T0][YB42AC175883-000616AB076854CB-0-0] [lt=40][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713775585277268}) [2024-04-22 16:46:25.283914] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.284039] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=16397929, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=179, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:25.317066] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.317122] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=37000, remain_us=1167664, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=37, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.327313] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:46:25.327372] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=53][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:25.327410] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775585057578708, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775585327299) [2024-04-22 16:46:25.327434] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775585127179, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:25.358024] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.358116] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=38000, remain_us=1126670, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=38, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.362178] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=42] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:46:25.363109] INFO [LIB] stat (utility.h:1165) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=4980, this=0x7f4d7e49f330, extra_info="palf_id:1") [2024-04-22 16:46:25.363170] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=58] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.131:2882", has_set_deleted:false}, accum_size=854) [2024-04-22 16:46:25.363229] INFO [LIB] stat (utility.h:1182) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=35] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=76, this=0x7f4d95343700, extra_info="") [2024-04-22 16:46:25.363352] INFO [LIB] stat (utility.h:1165) [13927][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=60] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=69, this=0x7f4d7e49f560, extra_info="palf_id:1") [2024-04-22 16:46:25.364398] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E69-0-0] [lt=35] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=41, this=0x7f4d7e4956a0, extra_info="palf_id:1") [2024-04-22 16:46:25.364452] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E69-0-0] [lt=52] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=34124, this=0x7f4d7e4958d0, extra_info="palf_id:1") [2024-04-22 16:46:25.364480] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E69-0-0] [lt=25] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.131:2882", total_slide_log_cnt=7, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=22, avg_log_submit_to_flush_time=5147, avg_log_submit_to_slide_time=34102) [2024-04-22 16:46:25.364495] INFO [PALF] read (log_cache.cpp:100) [14084][T1_ReplaySrv5][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=854, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2024-04-22 16:46:25.364744] INFO [LIB] stat (utility.h:1165) [14080][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=21] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=89, this=0x7f4d98c048f8, extra_info="") [2024-04-22 16:46:25.364805] INFO [LIB] stat (utility.h:1165) [14080][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=59] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=16, this=0x7f4d98c04b28, extra_info="") [2024-04-22 16:46:25.387986] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.388074] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=16293855, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=180, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:25.400370] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.400490] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=39000, remain_us=1084296, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=39, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.426310] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=143] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20814036992, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:25.442970] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.443043] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=40000, remain_us=1041743, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=40, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.486979] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.487044] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=41000, remain_us=997743, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=41, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.491924] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.492030] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=16189899, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=181, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:25.527568] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:25.527673] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=102][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775585357901873, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775585527553) [2024-04-22 16:46:25.527700] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775585327453, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:25.528185] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:25.531949] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.532009] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=42000, remain_us=952777, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=42, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.549760] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [13778][TsMgr][T0][Y0-0000000000000000-0-0] [lt=38] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:46:25.554844] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=17][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:25.554907] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=51] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:25.554951] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=29] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:46:25.554986] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=30][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:46:25.555014] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=25][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:46:25.555036] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=21][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:46:25.555053] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=12] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:25.555066] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=11][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:25.555079] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:25.555097] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:46:25.555110] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:25.555121] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:25.555156] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=32][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:25.555174] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=16][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:25.555203] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=27][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:25.555222] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=16][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:25.555240] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=16][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:25.555272] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=19][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:25.555290] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=14][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:25.555306] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:25.555319] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:25.555330] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-04-22 16:46:25.555357] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:25.555381] WDIAG [SERVER] do_process (obmp_query.cpp:760) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=23][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-04-22 16:46:25.555418] WDIAG [SERVER] do_process (obmp_query.cpp:884) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=31][errcode=-5019] query failed(ret=-5019, session={this:0x7f4d7edd01f8, id:3221225477, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-04-22 16:46:25.555464] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673527-0-0] [lt=30] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0x579c808 0x578fbfb 0x5788b5f 0x577a048 0xabd78a4 0x16c398be 0x7f4dc8ea7ea5 0x7f4dc8bd0b0d") [2024-04-22 16:46:25.573713] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=51] Cache replace map node details(ret=0, replace_node_count=0, replace_time=11349, replace_start_pos=2579474, replace_num=62914) [2024-04-22 16:46:25.573742] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:46:25.577695] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.577786] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=43000, remain_us=907000, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=43, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.595520] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.595605] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=16086324, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=182, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:25.626758] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.626837] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=44000, remain_us=857949, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=44, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.627125] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=63] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20814036992, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:25.627668] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:46:25.674597] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.674734] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=45000, remain_us=810052, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=45, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.699809] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.699903] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=15982026, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=183, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:25.723855] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.723943] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=46000, remain_us=760844, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=46, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.727783] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=56][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:25.727891] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=76][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775585548587145, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775585727768) [2024-04-22 16:46:25.727922] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775585527717, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:25.739560] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC35-0-0] [lt=215][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:25.739720] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC35-0-0] [lt=154][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:25.739791] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC35-0-0] [lt=68][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:25.739859] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC35-0-0] [lt=64][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:25.743478] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [13622][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=82][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:455, tid:13686}]) [2024-04-22 16:46:25.746670] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [14061][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=31] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-04-22 16:46:25.746732] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [14061][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=43] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=381) [2024-04-22 16:46:25.747099] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [13617][ServerTracerTim][T0][YB42AC175883-000616AB07F733A1-0-0] [lt=15][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-04-22 16:46:25.747122] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [13617][ServerTracerTim][T0][YB42AC175883-000616AB07F733A1-0-0] [lt=22][errcode=-4002] refresh sever list failed(ret=-4002) [2024-04-22 16:46:25.747131] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [13617][ServerTracerTim][T0][YB42AC175883-000616AB07F733A1-0-0] [lt=9][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-04-22 16:46:25.764654] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:306) [14058][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=12] ====== traversal_flush timer task ====== [2024-04-22 16:46:25.764817] INFO [MDS] flush (mds_table_mgr.cpp:172) [14058][T1_Flush][T1][YB42AC175883-000616AB087733A0-0-0] [lt=49] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f4d7e454510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713775597210958129, latest_log_ts:{val:1713775580545744419, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x7f4d953f4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2024-04-22 16:46:25.765011] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:338) [14058][T1_Flush][T1][YB42AC175883-000616AB087733A0-0-0] [lt=179] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-04-22 16:46:25.770129] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=26][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:46, local_retry_times:46, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:25.770232] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=80][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:25.770254] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:25.770265] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:25.770316] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=35][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:25.773875] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=9) [2024-04-22 16:46:25.773991] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=26][errcode=-4725] [2024-04-22 16:46:25.774013] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:25.774034] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:25.774049] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:25.774063] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:25.774075] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:25.774086] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:25.774140] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.774260] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=110][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:25.774285] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=22][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:25.774323] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=35][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=47, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:25.774390] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=49] will sleep(sleep_us=47000, remain_us=710397, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=47, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.800074] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=38][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:183, local_retry_times:183, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:25.800159] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:25.800180] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:25.800192] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:25.800218] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:25.804094] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] [2024-04-22 16:46:25.804164] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=67][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:25.804187] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=21][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:25.804202] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:25.804216] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:25.804261] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=42][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:25.804276] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:25.804363] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.804420] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=50][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:25.804442] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:25.804463] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=184, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:25.804497] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=18] will sleep(sleep_us=100000, remain_us=15877433, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=184, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:25.821580] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=37][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:47, local_retry_times:47, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:25.821769] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=148][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:25.821792] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:25.821806] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:25.821860] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=17][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:25.825145] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=29][errcode=-4725] [2024-04-22 16:46:25.825192] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=45][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:25.825225] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:25.825243] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:25.825257] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:25.825290] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=31][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:25.825301] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:25.825350] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.825389] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:25.825402] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:25.825420] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=48, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:25.825454] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19] will sleep(sleep_us=48000, remain_us=659332, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=48, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.827830] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=80] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20814036992, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:25.873661] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:48, local_retry_times:48, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:25.873751] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=68][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:25.873780] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:25.873794] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:25.873843] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=34][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:25.877266] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] [2024-04-22 16:46:25.877314] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=45][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:25.877347] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:25.877366] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=17][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:25.877381] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:25.877393] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:25.877404] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:25.877450] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.877490] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=33][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:25.877504] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:25.877544] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=37][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=49, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:25.877579] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19] will sleep(sleep_us=49000, remain_us=607208, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=49, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.892358] INFO [COMMON] handle (memory_dump.cpp:468) [13638][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-04-22 16:46:25.901180] INFO handle (memory_dump.cpp:589) [13638][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=61] statistics: tenant_cnt: 4, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 115 208 0 1 3 1 1 0 1 4 1 1 0 1 5 1 4 0 1 7 1 2 0 1 8 53 1 0 1 12 1 2 0 1 14 1 1 0 1 18 2 2 0 500 0 53 213 0 500 7 3 3 0 500 8 55 1 0 500 9 4 1 0 500 10 7 2 0 500 16 1 1 0 500 17 12 7 0 500 22 3 48 0 500 23 16 10 0 508 0 3 8 0 508 8 8 1 0 509 0 3 8 0 509 8 5 1 0 cost_time: 8635 [2024-04-22 16:46:25.904667] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:184, local_retry_times:184, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:25.904727] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=42][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:25.904748] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:25.904759] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:25.904799] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=28][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:25.908459] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] [2024-04-22 16:46:25.908483] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=23][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:25.908502] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=17][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:25.908516] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:25.908528] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:25.908537] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=8][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:25.908544] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=7][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:25.908571] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=5] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.908594] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=20][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:25.908607] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:25.908617] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=185, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:25.908678] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=50] will sleep(sleep_us=100000, remain_us=15773251, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=185, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:25.926710] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:49, local_retry_times:49, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:25.926786] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=56][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:25.926820] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=32][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:25.926831] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:25.926865] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:25.928014] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:25.928085] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=67][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775585648998210, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775585928000) [2024-04-22 16:46:25.928124] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775585727957, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:25.930670] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] [2024-04-22 16:46:25.930696] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=24][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:25.930714] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:25.930725] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:25.930733] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:25.930740] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:25.930747] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=6][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:25.930775] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.930799] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:25.930819] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:25.930830] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=50, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:25.930851] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11] will sleep(sleep_us=50000, remain_us=553935, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=50, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:25.938336] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [14062][T1_LSArchiveMgr][T1][YB42AC175883-000616AB07773224-0-0] [lt=32] gc stale ls task succ [2024-04-22 16:46:25.939025] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:222) [13719][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=47] [ASYNC TASK QUEUE](queue_size=0) [2024-04-22 16:46:25.940170] INFO do_work (ob_rl_mgr.cpp:704) [13683][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=45] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:46:25.941711] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [14055][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=52] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=13280, mem_used=68631552) [2024-04-22 16:46:25.942516] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [14072][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=58] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:46:25.956222] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [14014][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-04-22 16:46:25.956994] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=18][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:25.957106] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=21][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:25.957093] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=78] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:46:25.957126] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:46:25.957146] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:46:25.957128] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=13] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-04-22 16:46:25.957168] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=37][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, ret=-5019) [2024-04-22 16:46:25.957166] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:46:25.957177] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=8][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_stat, db_name=oceanbase) [2024-04-22 16:46:25.957187] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=9][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_stat) [2024-04-22 16:46:25.957184] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:25.957196] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=6][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:25.957197] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:25.957202] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=6][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:25.957212] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=6][errcode=-5019] Table 'oceanbase.__all_sys_stat' doesn't exist [2024-04-22 16:46:25.957219] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=5][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:25.957215] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:46:25.957224] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:25.957227] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:25.957239] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:25.957243] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=6][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:25.957251] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=6][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:25.957249] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=9][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:25.957258] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=5][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:25.957268] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:25.957275] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=6][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:25.957260] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:25.957289] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=7][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:25.957286] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=24][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:25.957296] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=5][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:25.957298] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:25.957304] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=6][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:25.957311] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=6][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:25.957309] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:25.957317] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=5][errcode=-5019] fail to handle text query(stmt=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2024-04-22 16:46:25.957325] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=7][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:25.957332] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:25.957332] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:25.957344] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14014][T1_ReqMemEvict][T1][YB42AC175883-000616AB089733A0-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:25.957345] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13614][ServerGTimer][T1][YB42AC175883-000616AB08073460-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:25.961289] WDIAG load_file_to_string (utility.h:627) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=0][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:46:25.961368] WDIAG get_ethernet_speed (utility.cpp:580) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=62][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:46:25.961399] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:46:25.961425] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:46:25.974104] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=105] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=8) [2024-04-22 16:46:25.980974] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:50, local_retry_times:50, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:25.981061] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=67][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:25.981090] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:25.981104] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:25.981130] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:25.985014] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] [2024-04-22 16:46:25.985090] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=73][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:25.985124] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=31][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:25.985143] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:25.985158] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:25.985170] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:25.985181] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:25.985228] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:25.985285] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=48][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:25.985307] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:25.985328] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=51, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:25.985362] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19] will sleep(sleep_us=51000, remain_us=499424, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=51, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:26.008903] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:185, local_retry_times:185, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:26.008994] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=70][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.009023] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.009043] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:26.009100] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:26.013107] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=22][errcode=-4725] [2024-04-22 16:46:26.013156] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=46][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:26.013189] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:26.013214] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:26.013259] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=42][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:26.013278] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=17][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:26.013292] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.013339] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.013380] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=33][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.013393] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:26.013411] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=186, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:26.013444] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=18] will sleep(sleep_us=100000, remain_us=15668485, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=186, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:26.028479] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=51] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20814036992, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:26.036667] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=45][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:51, local_retry_times:51, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:26.036756] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=68][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.036785] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.036798] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:26.036851] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:26.040142] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=29][errcode=-4725] [2024-04-22 16:46:26.040180] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=35][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:26.040209] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:26.040232] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:26.040248] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:26.040259] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:26.040270] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.040339] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=33] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.040393] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=46][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.040415] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:26.040435] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=52, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:26.040468] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19] will sleep(sleep_us=52000, remain_us=444318, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=52, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:26.046798] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13793][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.047434] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13802][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.047692] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13779][RpcIO][T0][Y0-0000000000000000-0-0] [lt=57] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.047708] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13801][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=38] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.047704] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13797][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=37] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.047812] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13806][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.047862] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13792][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.047885] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13799][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=50] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.047964] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13800][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=44] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.048069] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13780][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.048196] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13787][BatchIO][T0][Y0-0000000000000000-0-0] [lt=38] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.048267] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13783][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.048337] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:131) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14] ====== checkpoint timer task ====== [2024-04-22 16:46:26.048322] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13794][BatchIO][T0][Y0-0000000000000000-0-0] [lt=54] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.048376] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:748) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=20] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2024-04-22 16:46:26.048428] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1023) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=24] get_min_unreplayed_log_info(lsn={lsn:1377561}, scn={val:1713775585951605313, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713775585969867, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:7851, ret_code_:0}}, next_to_submit_lsn_:{lsn:1377561}, next_to_submit_scn_:{val:1713775585951605313, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f4d7c89f000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:1377439}, end_lsn:{lsn:1377561}, read_buf:{buf_len_:8192, buf_:0x7f4d7c89f000}, block_size:67104768, log_storage_:0x7f4d7e49b4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713775585951605312, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713775585951605312, v:0}, data_checksum:2923900223, flag:0}}, init_mode_version:0, accumulate_checksum:586469134, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713775585951605312, v:0}}}}}) [2024-04-22 16:46:26.048739] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13791][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.048802] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13785][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.048881] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13790][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.049139] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13788][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.049201] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13796][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.049201] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13798][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=30] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.049375] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13782][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.049512] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1426) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=57] succ to get rec scn(*this={this:0x7f4d7c604030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1713773681663296, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:46:26.049554] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:180) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=41] tx ctx memtable get rec scn(this={ObITable:{this:0x7f4d7e532080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f4d7e532080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-04-22 16:46:26.049601] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:856) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=38] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-04-22 16:46:26.049690] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13784][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.049705] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13804][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.049682] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:259) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=53] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f4d7e454510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713775597210958129, latest_log_ts:{val:1713775580545744419, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x7f4d953f4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2024-04-22 16:46:26.049755] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:599) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=69] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1}) [2024-04-22 16:46:26.049786] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:470) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=23] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2024-04-22 16:46:26.049803] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=13] get rec log scn(service_type_=0, rec_log_ts={val:1713775580545744419, v:0}) [2024-04-22 16:46:26.049819] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-04-22 16:46:26.049829] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:306) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=2, rec_log_ts={val:1713773717210958135, v:0}) [2024-04-22 16:46:26.049848] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:164) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1713773717210958135, v:0}, checkpoint_scn_in_ls_meta={val:1713773717210958135, v:0}, ls_id={id:1}, service_type="DAS_ID") [2024-04-22 16:46:26.049870] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1579) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"172.23.88.131:2882", has_set_deleted:false}, curr_base_lsn={lsn:0}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2024-04-22 16:46:26.049888] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:185) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=17] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2024-04-22 16:46:26.049903] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:192) [14057][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-04-22 16:46:26.050098] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13781][RpcIO][T0][Y0-0000000000000000-0-0] [lt=52] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.050397] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13789][BatchIO][T0][Y0-0000000000000000-0-0] [lt=49] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:26.064797] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397A-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:26.064873] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397A-0-0] [lt=51] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:46:26.064938] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397A-0-0] [lt=57][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:26.092740] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=38][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:52, local_retry_times:52, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:26.092827] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.092868] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=38][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.092899] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=28][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:26.092943] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=28][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:26.096694] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=22][errcode=-4725] [2024-04-22 16:46:26.096741] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=44][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:26.096783] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=39][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:26.096817] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=29][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:26.096849] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=29][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:26.096864] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:26.096875] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.096964] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=46] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.097040] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=65][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.097073] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=27][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:26.097123] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=46][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=53, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:26.097195] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=53] will sleep(sleep_us=53000, remain_us=387592, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=53, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:26.113699] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:186, local_retry_times:186, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:26.113786] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.113823] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=34][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.113851] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=26][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:26.113920] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=53][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:26.117484] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=37][errcode=-4725] [2024-04-22 16:46:26.117535] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=48][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:26.117576] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=37][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:26.117691] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=31][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:26.117726] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=32][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:26.117755] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=27][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:26.117769] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.117815] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.117872] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=48][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.117903] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=27][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:26.117923] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=187, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:26.117992] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=54] will sleep(sleep_us=100000, remain_us=15563938, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=187, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:26.127939] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=83] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:46:26.128256] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:26.128327] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=67][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775585951605312, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775586128241) [2024-04-22 16:46:26.128359] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775585928148, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:26.150341] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=42][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:53, local_retry_times:53, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:26.150385] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=33][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.150401] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.150413] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:26.150431] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:26.153819] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=25][errcode=-4725] [2024-04-22 16:46:26.153846] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=25][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:26.153866] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:26.153881] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:26.153891] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:26.153900] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:26.153907] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.153937] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.153963] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=22][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.153972] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:26.153983] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=54, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:26.154005] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12] will sleep(sleep_us=54000, remain_us=330781, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=54, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:26.159055] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7BE-0-0] [lt=55] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x7f4d7e495470, extra_info="") [2024-04-22 16:46:26.159162] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7BE-0-0] [lt=73] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.131:2882", role="FOLLOWER", total_group_log_cnt=7, avg_log_batch_cnt=0, total_group_log_size=854, avg_group_log_size=122) [2024-04-22 16:46:26.159198] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7BE-0-0] [lt=32] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.131:2882", committed size=854) [2024-04-22 16:46:26.168316] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1172) [14085][T1_ReplaySrv6][T1][Y0-0000000000000000-0-0] [lt=32] succ to submit log task to replay service(to_submit_lsn={lsn:1377617}, to_submit_scn={val:1713775586153821956, v:0}, replay_status={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1713775586168206, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:7851, ret_code_:0}}, next_to_submit_lsn_:{lsn:1377683}, next_to_submit_scn_:{val:1713775586153821957, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:0x7f4d7c89f000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:1377561}, end_lsn:{lsn:1377683}, read_buf:{buf_len_:8192, buf_:0x7f4d7c89f000}, block_size:67104768, log_storage_:0x7f4d7e49b4f0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1713775586153821956, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1713775586153821956, v:0}, data_checksum:2923900223, flag:1}}, init_mode_version:0, accumulate_checksum:420499129, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1713775586153821956, v:0}}}}}) [2024-04-22 16:46:26.174322] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=62] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=7) [2024-04-22 16:46:26.185830] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [13693][LSMetaTblUp0][T0][YB42AC175883-000616AB08B9F871-0-0] [lt=46][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713775570633812}) [2024-04-22 16:46:26.200584] INFO eloop_run (eloop.c:140) [13667][pnio1][T0][Y0-0000000000000000-0-0] [lt=59] PNIO [ratelimit] time: 1713775586200580, bytes: 91655830, bw: 0.019742 MB/s, add_ts: 1007575, add_bytes: 20858 [2024-04-22 16:46:26.208147] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=31][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:54, local_retry_times:54, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:26.208230] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=62][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.208252] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.208264] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:26.208314] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:26.211787] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] [2024-04-22 16:46:26.211838] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=48][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:26.211867] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:26.211885] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:26.211898] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:26.211909] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:26.211939] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=28][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.212014] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.212070] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=48][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.212096] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=22][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:26.212124] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=26][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=55, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:26.212180] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=38] will sleep(sleep_us=55000, remain_us=272607, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=55, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:26.214695] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173932-0-0] [lt=16] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:18, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713775573499769, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"]) [2024-04-22 16:46:26.216961] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173932-0-0] [lt=292][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:26.217036] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173932-0-0] [lt=55] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:46:26.217068] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173932-0-0] [lt=28][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:26.218218] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=78][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:187, local_retry_times:187, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:26.218302] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.218329] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.218346] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:26.218409] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:26.221939] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=50][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:26.222060] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.222186] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=15459744, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=188, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:26.227552] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:213) [14040][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=32] start to check and clear obj lock(ls->get_ls_id()={id:1}) [2024-04-22 16:46:26.227758] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [13953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=131] ====== tenant freeze timer task ====== [2024-04-22 16:46:26.229226] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=93] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20814036992, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:26.235328] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC38-0-0] [lt=383][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:26.235399] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC38-0-0] [lt=66][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:26.235473] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.235521] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.235469] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC38-0-0] [lt=67][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:26.235549] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.235571] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.235552] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC38-0-0] [lt=80][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:26.235593] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.235613] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.235693] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=77] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.235738] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.235760] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.235781] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.235803] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.235824] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.235845] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.235866] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.235887] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.235908] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:26.245065] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [13622][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=59][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:109, tid:13614}]) [2024-04-22 16:46:26.271322] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.271411] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=56000, remain_us=213376, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=56, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:26.272139] INFO eloop_run (eloop.c:140) [13674][pnio2][T0][Y0-0000000000000000-0-0] [lt=47] PNIO [ratelimit] time: 1713775586272135, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007629, add_bytes: 0 [2024-04-22 16:46:26.326246] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.326342] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=15355588, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=189, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:26.326311] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [13720][SysLocAsyncUp0][T0][YB42AC175883-000616AB076854EB-0-0] [lt=55][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713775586326211}) [2024-04-22 16:46:26.328544] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:46:26.328714] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=161][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:26.328817] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775586153821956, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775586328527) [2024-04-22 16:46:26.328883] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775586128379, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:26.331700] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.331814] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=57000, remain_us=152972, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=57, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:26.365771] INFO [LIB] stat (utility.h:1165) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=24] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=6326, this=0x7f4d7e49f330, extra_info="palf_id:1") [2024-04-22 16:46:26.365845] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=63] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.131:2882", has_set_deleted:false}, accum_size=732) [2024-04-22 16:46:26.365906] INFO [LIB] stat (utility.h:1182) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=36] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=66, this=0x7f4d95343700, extra_info="") [2024-04-22 16:46:26.366025] INFO [LIB] stat (utility.h:1165) [13927][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=90, this=0x7f4d7e49f560, extra_info="palf_id:1") [2024-04-22 16:46:26.369498] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E6F-0-0] [lt=17] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=26, this=0x7f4d7e4956a0, extra_info="palf_id:1") [2024-04-22 16:46:26.369568] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E6F-0-0] [lt=67] [PALF STAT LOG LIFE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=8093, this=0x7f4d7e4958d0, extra_info="palf_id:1") [2024-04-22 16:46:26.369546] INFO [PALF] read (log_cache.cpp:100) [14084][T1_ReplaySrv5][T1][Y0-0000000000000000-0-0] [lt=53] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=732, hit_cnt=6, read_cnt=6, hit rate=1.000000000000000000e+00) [2024-04-22 16:46:26.369594] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E6F-0-0] [lt=23] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.131:2882", total_slide_log_cnt=6, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=26, avg_log_submit_to_flush_time=6492, avg_log_submit_to_slide_time=8067) [2024-04-22 16:46:26.369746] INFO [LIB] stat (utility.h:1165) [14080][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=26] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=127, this=0x7f4d98c048f8, extra_info="") [2024-04-22 16:46:26.369793] INFO [LIB] stat (utility.h:1165) [14080][T1_ReplaySrv1][T1][Y0-0000000000000000-0-0] [lt=46] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=10, this=0x7f4d98c04b28, extra_info="") [2024-04-22 16:46:26.374508] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=56] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=6) [2024-04-22 16:46:26.392784] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.392885] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=58000, remain_us=91901, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=58, v.err_=-4725, timeout_timestamp=1713775586484785) [2024-04-22 16:46:26.430226] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=110] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:26.430529] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.430609] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=15251321, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=190, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:26.453265] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1][errcode=-4012] [2024-04-22 16:46:26.453324] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=56][errcode=-4012] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4012, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:26.453357] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=30][errcode=-4012] failed to setup first scanner(ret=-4012) [2024-04-22 16:46:26.453373] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4012] Open this operator failed(ret=-4012, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:26.453388] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4012] open operator failed(ret=-4012) [2024-04-22 16:46:26.453418] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=28][errcode=-4012] fail open main query(ret=-4012) [2024-04-22 16:46:26.453431] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:46:26.453442] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:46:26.453454] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:26.453473] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=59, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:26.453497] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:59, local_retry_times:59, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-04-22 16:46:26.453529] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=22][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:46:26.453540] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:46:26.453550] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-04-22 16:46:26.453571] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-04-22 16:46:26.453590] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=15] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1968325) [2024-04-22 16:46:26.453685] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=93][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-04-22 16:46:26.453708] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:46:26.453723] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-04-22 16:46:26.453736] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:26.453751] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-04-22 16:46:26.453774] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:46:26.453877] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-04-22 16:46:26.453920] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=40][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:46:26.453946] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=24][errcode=-4012] get failed(ret=-4012) [2024-04-22 16:46:26.453964] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:888) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:46:26.454021] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4507) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=52][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:46:26.454043] WDIAG [SHARE.SCHEMA] operator() (ob_multi_version_schema_service.cpp:2425) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4012] fail to get baseline_schema_version(ret=-4012) [2024-04-22 16:46:26.454056] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2495) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=10] [REFRESH_SCHEMA] end refresh and add schema(ret=-4012, ret="OB_TIMEOUT", tenant_ids=[], cost=1969279) [2024-04-22 16:46:26.454077] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:308) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4012] fail to refresh and add schema(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:46:26.454089] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:314) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=11] [REFRESH_SCHEMA] end refresh schema with new mode, (ret=-4012, ret="OB_TIMEOUT", tenant_ids=[], used time=1969332) [2024-04-22 16:46:26.457981] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.458096] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] already timeout, do not need sleep(sleep_us=0, remain_us=1996018, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.460856] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.460942] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=1000, remain_us=1993172, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.464691] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.464792] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=2000, remain_us=1989322, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.469236] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.469311] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=3000, remain_us=1984803, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.474884] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.474960] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=4000, remain_us=1979154, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.477370] INFO [CLOG] operator() (ob_log_replay_service.cpp:1492) [14030][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=16] get_replay_process success(id={id:1}, replayed_log_size=1377927, unreplayed_log_size=0) [2024-04-22 16:46:26.477483] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:156) [14030][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=57] dump tenant replay process(tenant_id=1, unreplayed_log_size(MB)=0, estimate_time(second)=0, replayed_log_size(MB)=1, last_replayed_log_size(MB)=1, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2024-04-22 16:46:26.481410] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.481493] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=5000, remain_us=1972621, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.489512] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.489598] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=6000, remain_us=1964517, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.499871] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.499974] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=7000, remain_us=1954140, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.510804] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.510884] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=8000, remain_us=1943230, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.522668] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.522772] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=9000, remain_us=1931343, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.528741] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:26.528812] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=66][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775586353530212, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775586528726) [2024-04-22 16:46:26.528842] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775586328920, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:26.532023] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=33][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:26.533494] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.533577] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=15148352, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=191, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:26.535763] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.535843] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=10000, remain_us=1918271, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.549727] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.549862] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=11000, remain_us=1904252, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.552567] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [13778][TsMgr][T0][Y0-0000000000000000-0-0] [lt=51] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1713775586051108], gts:1713775586052531004, latest_srr:[mts=1713775586552491]}) [2024-04-22 16:46:26.552694] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [13778][TsMgr][T0][Y0-0000000000000000-0-0] [lt=123] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:46:26.553319] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [13667][pnio1][T0][YB42AC175883-000616AB095740C3-0-0] [lt=113] ts response task statistics(alloc_count=14350, free_count=7174) [2024-04-22 16:46:26.558742] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=24][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:26.558817] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=62] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:26.558865] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=30] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:46:26.558895] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:46:26.559133] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=231][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:46:26.559178] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=43][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:46:26.559214] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=19] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:26.559254] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=36][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:26.559272] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=17][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:26.559301] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=11][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:46:26.559317] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=13][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:26.559329] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:26.559341] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:26.559352] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:26.559365] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:26.559378] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:26.559390] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:26.559422] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=14][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:26.559439] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=12][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:26.559457] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=13][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:26.559470] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:26.559482] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-04-22 16:46:26.559513] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:26.559538] WDIAG [SERVER] do_process (obmp_query.cpp:760) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=23][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-04-22 16:46:26.559561] WDIAG [SERVER] do_process (obmp_query.cpp:884) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=15][errcode=-5019] query failed(ret=-5019, session={this:0x7f4d7edd01f8, id:3221225477, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-04-22 16:46:26.559609] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673528-0-0] [lt=31] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0x579c808 0x578fbfb 0x5788b5f 0x577a048 0xabd78a4 0x16c398be 0x7f4dc8ea7ea5 0x7f4dc8bd0b0d") [2024-04-22 16:46:26.565790] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.565895] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=12000, remain_us=1888219, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.568578] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [14021][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=84] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:46:26.574699] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=54] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=5) [2024-04-22 16:46:26.581952] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.582035] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=13000, remain_us=1872079, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.599584] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.599708] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=14000, remain_us=1854407, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.617955] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.618056] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=15000, remain_us=1836058, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.628837] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:46:26.631167] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=242] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20814036992, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:26.636791] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.636898] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=16000, remain_us=1817216, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.638895] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.638978] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=15042951, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=192, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:26.648310] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:189) [14116][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-04-22 16:46:26.648402] WDIAG [SERVER] do_heartbeat_event (ob_heartbeat.cpp:266) [14116][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=61][errcode=0] try reload schema failed(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=-4023) [2024-04-22 16:46:26.651263] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [13930][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=72] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):65, sequence:2}) [2024-04-22 16:46:26.656781] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.656888] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=17000, remain_us=1797226, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.677216] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.677256] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=18000, remain_us=1776858, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.698980] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.699013] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=19000, remain_us=1755100, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.722237] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.722277] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=20000, remain_us=1731837, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.728945] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:26.729015] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=65][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775586458148206, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775586728929) [2024-04-22 16:46:26.729071] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775586528863, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:26.731211] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC3B-0-0] [lt=273][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:26.731272] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC3B-0-0] [lt=57][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:26.731320] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC3B-0-0] [lt=45][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:26.731351] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC3B-0-0] [lt=30][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:26.743146] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.743235] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=14938695, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=193, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:26.747091] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.747123] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=21000, remain_us=1706991, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.771493] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.771526] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=22000, remain_us=1682588, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.774951] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=66] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=4) [2024-04-22 16:46:26.796885] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.796916] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=23000, remain_us=1657197, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.823549] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.823581] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=24000, remain_us=1630533, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.831916] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=96] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20814036992, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:26.845791] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.845876] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=14836053, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=194, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:26.846271] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [13622][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=80][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:520, tid:13686}]) [2024-04-22 16:46:26.846717] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=66] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=14731, clean_start_pos=1258290, clean_num=125829) [2024-04-22 16:46:26.847675] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:24, local_retry_times:24, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:26.847706] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=24][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.847716] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.847724] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:26.847744] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:26.849820] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] [2024-04-22 16:46:26.849835] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:26.849847] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:26.849856] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:26.849866] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:26.849874] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:26.849880] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=6][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.849905] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.849920] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.849926] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:26.849934] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=25, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:26.849948] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] will sleep(sleep_us=25000, remain_us=1604166, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.875088] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:25, local_retry_times:25, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:26.875188] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=80][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.875210] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.875221] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:26.875246] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:26.879156] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=46][errcode=-4725] [2024-04-22 16:46:26.879221] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=61][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:26.879260] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=35][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:26.879295] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=30][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:26.879327] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=28][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:26.879347] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:26.879361] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.879414] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.879464] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=43][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.879477] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:26.879495] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=26, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:26.879526] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18] will sleep(sleep_us=26000, remain_us=1574588, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.905673] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=65][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:26, local_retry_times:26, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:26.905705] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.905715] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.905722] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:26.905746] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:26.909316] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] [2024-04-22 16:46:26.909331] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:26.909343] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:26.909353] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:26.909361] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:26.909368] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=6][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:26.909374] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.909400] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.909416] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.909422] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:26.909430] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=27, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:26.909447] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8] will sleep(sleep_us=27000, remain_us=1544670, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.929188] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:26.929300] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=69][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775586755764037, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775586929172) [2024-04-22 16:46:26.929337] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775586729092, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:26.931240] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=26][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:26.931330] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=64] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:46:26.931383] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=46][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:46:26.931422] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=36][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:46:26.931445] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=21][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:46:26.931464] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:26.931477] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:26.931494] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:46:26.931507] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:26.931518] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:26.931529] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:26.931541] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:26.931553] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:26.931564] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:26.931576] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:26.931602] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:26.931615] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:26.931671] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=51][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:26.931684] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:26.931696] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='172.23.88.131' and svr_port=2882, ret=-5019) [2024-04-22 16:46:26.931711] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:26.931724] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.23.88.131' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:26.931773] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=43][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:26.931801] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=22][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:26.931812] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:26.931830] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=17][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:26.931853] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.23.88.131' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:26.931869] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13921][T1_Occam][T1][YB42AC175883-000616AB088733A2-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='172.23.88.131' and svr_port=2882"}, aret=-5019, ret=-5019) [2024-04-22 16:46:26.931884] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='172.23.88.131' and svr_port=2882) [2024-04-22 16:46:26.931898] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:26.931911] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:26.931924] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] query failed(ret=-5019, conn=0x7f4da0e3e050, start=1713775586931030, sql=SELECT zone FROM __all_server where svr_ip='172.23.88.131' and svr_port=2882) [2024-04-22 16:46:26.931941] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:26.931954] WDIAG get_my_sql_result_ (ob_table_access_helper.h:435) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f4d94a56298, table=__all_server, condition=where svr_ip='172.23.88.131' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='172.23.88.131' and svr_port=2882, columns_str="zone") [2024-04-22 16:46:26.932000] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:332) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='172.23.88.131' and svr_port=2882) [2024-04-22 16:46:26.932103] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:635) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f4d94a56298, where_condition="where svr_ip='172.23.88.131' and svr_port=2882", zone_name_holder=) [2024-04-22 16:46:26.932122] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:568) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-04-22 16:46:26.932146] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:577) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-04-22 16:46:26.932160] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:144) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:26.936530] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:27, local_retry_times:27, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:26.936560] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.936569] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.936576] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:26.936590] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:26.938457] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [14062][T1_LSArchiveMgr][T1][YB42AC175883-000616AB07773224-0-0] [lt=46] gc stale ls task succ [2024-04-22 16:46:26.939101] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [13752][Blacklist][T0][Y0-0000000000000000-0-0] [lt=58] blacklist_loop exec finished(cost_time=73, is_enabled=true, send_cnt=0) [2024-04-22 16:46:26.940021] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] [2024-04-22 16:46:26.940037] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:26.940049] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:26.940058] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:26.940067] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:26.940075] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:26.940083] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.940104] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.940120] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.940126] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:26.940134] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=28, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:26.940158] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16] will sleep(sleep_us=28000, remain_us=1513956, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.940310] INFO do_work (ob_rl_mgr.cpp:704) [13683][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=31] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:46:26.940324] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [14156][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=44] current monitor number(seq_=-1) [2024-04-22 16:46:26.941901] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [14055][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=86] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=13333, mem_used=68631552) [2024-04-22 16:46:26.942762] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [14072][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=66] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:46:26.946019] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:194, local_retry_times:194, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:26.946112] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=74][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.946130] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=17][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.946139] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=8][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:26.946188] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:26.949549] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=23][errcode=-4725] [2024-04-22 16:46:26.949596] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=44][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:26.949704] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=104][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:26.949721] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:26.949735] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:26.949747] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:26.949758] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.949808] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.949848] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=33][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.949860] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:26.949878] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=195, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:26.949911] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=18] will sleep(sleep_us=100000, remain_us=14732019, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=195, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:26.961872] WDIAG load_file_to_string (utility.h:627) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:46:26.961917] WDIAG get_ethernet_speed (utility.cpp:580) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:46:26.961937] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:46:26.961954] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:46:26.968239] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:28, local_retry_times:28, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:26.968271] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=23][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.968293] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:26.968300] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=6][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:26.968321] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=6][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:26.971332] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] [2024-04-22 16:46:26.971347] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:26.971359] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:26.971376] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:26.971385] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:26.971392] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=6][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:26.971397] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.971418] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:26.971435] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:26.971440] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=4][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:26.971449] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=29, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:26.971465] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8] will sleep(sleep_us=29000, remain_us=1482649, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:26.975108] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=61] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=3) [2024-04-22 16:46:27.000550] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:29, local_retry_times:29, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:27.000582] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=24][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.000599] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.000607] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:27.000634] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:27.004391] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] [2024-04-22 16:46:27.004405] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:27.004416] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:27.004425] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:27.004434] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:27.004442] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:27.004449] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.004467] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=6] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.004483] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.004489] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:27.004505] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=30, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.004520] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] will sleep(sleep_us=30000, remain_us=1449594, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.028581] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:237) [13917][T1_Occam][T1][YB42AC175883-000616AB08273819-0-0] [lt=89] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-04-22 16:46:27.034606] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:30, local_retry_times:30, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:27.034652] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=38][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.034662] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.034669] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:27.034692] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:27.037512] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] [2024-04-22 16:46:27.037527] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:27.037539] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:27.037555] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:27.037564] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:27.037570] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=6][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:27.037576] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.037596] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=4] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.037619] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.037643] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=23][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:27.037651] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=31, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.037666] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] will sleep(sleep_us=31000, remain_us=1416447, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.047098] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13799][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=41] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.047152] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13802][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=48] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.047157] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13801][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=72] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.047364] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13787][BatchIO][T0][Y0-0000000000000000-0-0] [lt=97] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.047397] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20814036992, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:27.047698] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13806][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.047839] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13779][RpcIO][T0][Y0-0000000000000000-0-0] [lt=53] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.047943] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13793][BatchIO][T0][Y0-0000000000000000-0-0] [lt=53] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.047952] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13797][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=134] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.047971] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13794][BatchIO][T0][Y0-0000000000000000-0-0] [lt=38] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.048028] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13800][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=27] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.048228] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13780][RpcIO][T0][Y0-0000000000000000-0-0] [lt=79] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.048309] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13798][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=27] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.048374] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13783][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.048525] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13791][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.048753] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13785][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.048917] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13788][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.048919] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13790][BatchIO][T0][Y0-0000000000000000-0-0] [lt=66] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.048953] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13792][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.049069] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13796][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=37] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.049375] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13784][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.049429] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13782][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.049805] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13804][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=30] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.049916] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13781][RpcIO][T0][Y0-0000000000000000-0-0] [lt=37] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.050025] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=26][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:195, local_retry_times:195, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:27.050063] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=26][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.050093] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=29][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.050103] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:27.050120] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=8][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:27.050517] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13789][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:27.053509] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] [2024-04-22 16:46:27.053558] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=47][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:27.053581] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=20][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:27.053594] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:27.053605] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:27.053667] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=60][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:27.053677] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.053715] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.053753] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=33][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.053780] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=25][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:27.053795] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=14][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=196, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.053822] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16] will sleep(sleep_us=100000, remain_us=14628107, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=196, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:27.067574] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=0][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:27.067698] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=99] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:46:27.067734] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=30][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:46:27.067763] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=26][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:46:27.067793] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=27][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:46:27.067835] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=30][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:27.067871] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=33][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:27.067960] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=79][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:46:27.067976] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=13][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:27.067988] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:27.068000] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=11][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:27.068012] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=11][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:27.068025] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=12][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:27.068038] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:27.068050] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:27.068077] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:27.068092] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=12][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:27.068108] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=13][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:27.068125] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=15][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:27.068184] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=54][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:46:27.068213] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=26][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:27.068254] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=36][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.068289] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=28][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:27.068318] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=23][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:27.068330] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:27.068340] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=10][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:27.068365] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397B-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:27.068385] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397B-0-0] [lt=16][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:46:27.068410] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397B-0-0] [lt=22][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:27.068457] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397B-0-0] [lt=44][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:27.068477] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397B-0-0] [lt=18][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:27.068499] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397B-0-0] [lt=19][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7c0fc050, start=1713775587067305, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:27.068526] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397B-0-0] [lt=25][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:27.068539] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397B-0-0] [lt=10][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:46:27.068686] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397B-0-0] [lt=14][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:46:27.068724] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397B-0-0] [lt=35][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:46:27.068752] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:31, local_retry_times:31, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:27.068872] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=68][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.068909] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=34][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.068924] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:27.068950] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:27.072609] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=35][errcode=-4725] [2024-04-22 16:46:27.072691] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=80][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:27.072722] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=28][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:27.072762] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=37][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:27.072776] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:27.072796] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:27.072807] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.072858] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.072927] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=61][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.072950] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:27.072970] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=32, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.073003] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19] will sleep(sleep_us=32000, remain_us=1381111, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=32, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.105185] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=30][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:32, local_retry_times:32, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:27.105291] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=80][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.105345] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=50][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.105359] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:27.105384] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:27.109025] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] [2024-04-22 16:46:27.109069] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=41][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:27.109100] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:27.109117] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:27.109131] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:27.109142] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:27.109154] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.109198] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.109238] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=34][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.109268] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=28][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:27.109287] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=33, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.109320] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19] will sleep(sleep_us=33000, remain_us=1344794, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=33, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.129103] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:46:27.129434] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:27.129512] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=72][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775586957101054, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775587129419) [2024-04-22 16:46:27.129556] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775586929402, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:27.140475] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [14059][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=17] ====== check clog disk timer task ====== [2024-04-22 16:46:27.140532] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [14059][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=47] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=65) [2024-04-22 16:46:27.140584] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [14059][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=27] cannot_recycle_log_size statistics(cannot_recycle_log_size=1378284, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:46:27.142459] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:33, local_retry_times:33, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:27.142547] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.142573] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=23][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.142586] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:27.142665] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:27.146770] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=54][errcode=-4725] [2024-04-22 16:46:27.146808] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=35][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:27.146836] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:27.146858] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:27.146874] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:27.146885] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:27.146896] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.146941] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.146987] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=39][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.147000] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:27.147018] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=34, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.147070] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=36] will sleep(sleep_us=34000, remain_us=1307044, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=34, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.154000] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:196, local_retry_times:196, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:27.154126] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=68][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.154166] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=36][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.154186] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=18][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:27.154238] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=15][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:27.156693] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=20][errcode=-4725] [2024-04-22 16:46:27.156731] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=35][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:27.156759] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=25][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:27.156775] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:27.156788] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:27.156809] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:27.156820] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.156866] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.156919] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=45][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.156941] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:27.156966] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=22][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=197, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.157011] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=30] will sleep(sleep_us=100000, remain_us=14524918, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=197, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:27.160492] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7CA-0-0] [lt=91] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=0, this=0x7f4d7e495470, extra_info="") [2024-04-22 16:46:27.160584] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7CA-0-0] [lt=60] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.131:2882", role="FOLLOWER", total_group_log_cnt=6, avg_log_batch_cnt=0, total_group_log_size=723, avg_group_log_size=120) [2024-04-22 16:46:27.160673] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7CA-0-0] [lt=84] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.131:2882", committed size=723) [2024-04-22 16:46:27.175264] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=2) [2024-04-22 16:46:27.181204] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:34, local_retry_times:34, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:27.181303] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=75][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.181395] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=86][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.181431] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=33][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:27.181483] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=38][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:27.184722] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.184825] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=35000, remain_us=1269290, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=35, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.185754] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [13692][LSSysTblUp0][T0][YB42AC175883-000616AB08A9F863-0-0] [lt=98][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713775570633812}) [2024-04-22 16:46:27.200865] WDIAG [SQL] create_sessid (ob_sql_session_mgr.cpp:395) [13813][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=62][errcode=0] server is initiating(server_id=0, local_seq=1065, max_local_seq=262143, max_server_id=4095) [2024-04-22 16:46:27.200903] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [13813][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=35] new mysql sessid created(conn.sessid_=3221226537, support_ssl=false) [2024-04-22 16:46:27.201017] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [13813][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=66] sm conn init succ(conn.sessid_=3221226537, sess.client_addr_="127.0.0.1:49350") [2024-04-22 16:46:27.201046] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1019) [13813][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] accept one succ(*s={this:0x7f4d73604030, session_id:3221226537, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:244}, err:0, last_decode_time:0, last_write_time:1713775587201012, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:46:27.201128] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [13813][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=32] username and tenantname(user_name=ocp_monitor, tenant_name=) [2024-04-22 16:46:27.201173] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [13813][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=20] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-04-22 16:46:27.201192] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:280) [13813][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=17] mysql login queue(mysql_queue->queue_.size()=1) [2024-04-22 16:46:27.201219] INFO [SERVER] verify_connection (obmp_connect.cpp:1964) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=6] server is initializing, ignore verify_ip_white_list(status=1, ret=0) [2024-04-22 16:46:27.201326] INFO load_privilege_info (obmp_connect.cpp:545) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=27] no tenant name set, use default tenant name(tenant_name=sys) [2024-04-22 16:46:27.201983] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=22] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:27.202039] INFO [SHARE.SCHEMA] get_user_info (ob_schema_getter_guard.cpp:2213) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=49] user not exist(tenant_id=1, user_name=ocp_monitor) [2024-04-22 16:46:27.202051] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2620) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=11][errcode=-4043] No tenant user(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"127.0.0.1", db:"oceanbase", scramble_str:"}&=D{`NGZk/FSMiJr!J2"}, ret=-4043, ret="OB_PASSWORD_WRONG") [2024-04-22 16:46:27.202069] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:678) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=15][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"ocp_monitor", client_ip:"127.0.0.1", db:"oceanbase", scramble_str:"}&=D{`NGZk/FSMiJr!J2"}, ret=-4043) [2024-04-22 16:46:27.202080] WDIAG [SERVER] verify_identify (obmp_connect.cpp:2062) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=9][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-04-22 16:46:27.202090] WDIAG [SERVER] process (obmp_connect.cpp:346) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=8][errcode=-4043] fail to verify_identify(ret=-4043) [2024-04-22 16:46:27.202112] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=6] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0xaec6a47 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f4dc8ea7ea5 0x7f4dc8bd0b0d") [2024-04-22 16:46:27.202184] INFO [SERVER] free_session (obmp_base.cpp:326) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=19] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221226537, proxy_sessid:0}) [2024-04-22 16:46:27.202212] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:835) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=21][errcode=0] server close connection(sessid=3221226537, proxy_sessid=0, stack="0x5d1c83c 0xaeeb5ca 0xaeb47cd 0xaec6183 0x578ac98 0x175ba7a2 0x175bba1a 0x175bbf99 0x16c3d541 0x16c398be 0x7f4dc8ea7ea5 0x7f4dc8bd0b0d") [2024-04-22 16:46:27.202223] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221226537, proxy_sessid=0) [2024-04-22 16:46:27.202231] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:839) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=7][errcode=-4016] session is null [2024-04-22 16:46:27.202239] INFO [SERVER] process (obmp_connect.cpp:488) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=6] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=ocp_monitor, host_name=xxx.xxx.xxx.xxx, sessid=3221226537, proxy_sessid=0, client_sessid=3221226537, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762509, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-04-22 16:46:27.202269] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=28][errcode=-4018] get session fail(ret=-4018, sessid=3221226537, proxy_sessid=0) [2024-04-22 16:46:27.202275] WDIAG [SERVER] flush_buffer (obmp_packet_sender.cpp:1068) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=6][errcode=-4018] fail to get session info(ret=-4018) [2024-04-22 16:46:27.202296] WDIAG [SERVER] response (obmp_base.cpp:89) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=6][errcode=-4018] failed to flush_buffer(ret=-4018) [2024-04-22 16:46:27.202305] WDIAG [RPC.FRAME] run (ob_sql_processor.cpp:48) [14046][T1_MysqlQueueTh][T1][Y0-000616AB0857337E-0-0] [lt=8][errcode=-4018] response rpc result fail(ret=-4018) [2024-04-22 16:46:27.202361] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:825) [13813][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f4d73604030, session_id:3221226537, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:244}, err:5, last_decode_time:1713775587201124, last_write_time:1713775587202358, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:101, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-04-22 16:46:27.202406] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [13813][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=36] kill and revert session(conn.sessid_=3221226537, proxy_sessid=0, server_id=0, ret=0) [2024-04-22 16:46:27.202422] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:915) [13813][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f4d73604030, session_id:3221226537, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:244}, err:5, last_decode_time:1713775587201124, last_write_time:1713775587202358, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:101, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-04-22 16:46:27.202457] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [13813][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=31] connection close(sessid=3221226537, proxy_sessid=0, tenant_id=1, server_id=0, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:89, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-04-22 16:46:27.208207] INFO eloop_run (eloop.c:140) [13667][pnio1][T0][Y0-0000000000000000-0-0] [lt=53] PNIO [ratelimit] time: 1713775587208201, bytes: 91682906, bw: 0.025626 MB/s, add_ts: 1007621, add_bytes: 27076 [2024-04-22 16:46:27.218792] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=0] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:18, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713775573499769, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"]) [2024-04-22 16:46:27.220986] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=144][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:27.221058] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=52] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:46:27.221112] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=49][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:46:27.221132] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:46:27.221151] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:46:27.221170] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:27.221185] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=14][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:27.221213] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=18][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:46:27.221231] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=15][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:27.221248] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=16][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:27.221265] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=15][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:27.221285] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=18][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:27.221297] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:27.221322] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=23][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:27.221334] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:27.221360] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=14][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:27.221373] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:27.221395] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=18][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:27.221413] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=16][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:27.221480] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=64][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:46:27.221496] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=14][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:27.221509] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.221547] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=32][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:27.221665] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=31][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:27.221690] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:27.221725] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=32][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:27.221760] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=13][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:27.221778] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=17][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:46:27.221793] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:27.221807] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:27.221820] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=12][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:27.221834] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7c0e8050, start=1713775587220750, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:27.221851] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:27.221865] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:27.221955] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=14][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:46:27.221971] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=14][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:46:27.221995] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=22][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:27.222049] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=51][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:27.222067] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=14][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:46:27.222082] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=13][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:46:27.222094] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173933-0-0] [lt=11][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:27.224036] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.224136] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=36000, remain_us=1229979, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=36, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.229487] INFO [PALF] check_parent_health (log_config_mgr.cpp:2764) [13932][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=35] re_register_parent reason(palf_id=1, self="172.23.88.131:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1713775583224959, last_first_register_time_us=1713773690229985, register_time_us=1713775583224959, curr_time_us=1713775587229483) [2024-04-22 16:46:27.229607] INFO [PALF] check_parent_health (log_config_mgr.cpp:2769) [13932][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=56] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.23.88.131:2882") [2024-04-22 16:46:27.236965] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC3E-0-0] [lt=230][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:27.237038] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC3E-0-0] [lt=69][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:27.237087] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC3E-0-0] [lt=46][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:27.237145] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC3E-0-0] [lt=56][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:27.237500] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.237529] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.237545] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.237557] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.237568] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.237581] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.237589] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.237596] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.237667] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=69] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.237678] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.237686] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.237694] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.237701] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.237714] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.237721] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.237729] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:27.248079] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:27.248790] INFO [COMMON] wash (ob_kvcache_store.cpp:356) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=69] Wash time detail, (compute_wash_size_time=237, refresh_score_time=129, wash_time=0) [2024-04-22 16:46:27.257194] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=22][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:27.261336] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.261452] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=14420478, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=198, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:27.263993] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.264068] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=37000, remain_us=1190046, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=37, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.279747] INFO eloop_run (eloop.c:140) [13674][pnio2][T0][Y0-0000000000000000-0-0] [lt=54] PNIO [ratelimit] time: 1713775587279743, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007608, add_bytes: 0 [2024-04-22 16:46:27.305093] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.305184] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=38000, remain_us=1148930, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=38, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.329674] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:46:27.329733] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:27.329772] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775587160509302, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775587329659) [2024-04-22 16:46:27.329851] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=76][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775587129587, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:27.345941] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.346011] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [13720][SysLocAsyncUp0][T0][YB42AC175883-000616AB07685520-0-0] [lt=65][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713775587345910}) [2024-04-22 16:46:27.346033] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=39000, remain_us=1108081, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=39, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.365586] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.365777] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=14316153, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=199, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:27.375485] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=76] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=1) [2024-04-22 16:46:27.389290] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.389394] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=40000, remain_us=1064720, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=40, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.433415] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.433521] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=41000, remain_us=1020594, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=41, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.449474] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:27.469909] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.470006] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=14211923, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=200, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:27.478775] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.478859] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=42000, remain_us=975255, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=42, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.525004] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.525136] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=43000, remain_us=928979, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=43, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.529884] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:27.529983] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=65][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775587360536978, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775587529868) [2024-04-22 16:46:27.530178] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=186][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775587329875, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:27.555238] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [13778][TsMgr][T0][Y0-0000000000000000-0-0] [lt=40] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=150, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-04-22 16:46:27.555270] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [13778][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:46:27.562204] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=81][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:27.562273] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=56] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:27.562323] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=30] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:46:27.562393] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=64][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:46:27.562420] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=25][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:46:27.562439] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:46:27.562458] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=12] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:27.562471] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=11][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:27.562483] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:27.562503] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=11][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:46:27.562515] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:27.562527] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:27.562538] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:27.562549] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:27.562561] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:27.562584] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=21][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:27.562596] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:27.562621] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=14][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:27.562698] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=71][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:27.562721] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=18][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:27.562741] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=17][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:27.562756] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=13][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-04-22 16:46:27.562784] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:27.562811] WDIAG [SERVER] do_process (obmp_query.cpp:760) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=25][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-04-22 16:46:27.562833] WDIAG [SERVER] do_process (obmp_query.cpp:884) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=15][errcode=-5019] query failed(ret=-5019, session={this:0x7f4d7edd01f8, id:3221225477, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-04-22 16:46:27.562878] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [14106][T1_L0_G0][T1][YB42AC175883-000616AB08673529-0-0] [lt=29] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0x579c808 0x578fbfb 0x5788b5f 0x577a048 0xabd78a4 0x16c398be 0x7f4dc8ea7ea5 0x7f4dc8bd0b0d") [2024-04-22 16:46:27.565974] INFO [LIB] stat (utility.h:1165) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT WRITE LOG COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=4798, this=0x7f4d7e49f330, extra_info="palf_id:1") [2024-04-22 16:46:27.566038] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=60] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.131:2882", has_set_deleted:false}, accum_size=845) [2024-04-22 16:46:27.566116] INFO [LIB] stat (utility.h:1182) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=53] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=88, this=0x7f4d95343700, extra_info="") [2024-04-22 16:46:27.566332] INFO [LIB] stat (utility.h:1165) [13927][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=50] [PALF STAT FLUSH CB COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=93, this=0x7f4d7e49f560, extra_info="palf_id:1") [2024-04-22 16:46:27.567196] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175882-000616AB08DC3F8B-0-0] [lt=31] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=27, this=0x7f4d7e4956a0, extra_info="palf_id:1") [2024-04-22 16:46:27.567252] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175882-000616AB08DC3F8B-0-0] [lt=53] [PALF STAT LOG LIFE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=6131, this=0x7f4d7e4958d0, extra_info="palf_id:1") [2024-04-22 16:46:27.567280] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [17893][T1_L0_G1][T1][YB42AC175882-000616AB08DC3F8B-0-0] [lt=25] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.131:2882", total_slide_log_cnt=7, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=5027, avg_log_submit_to_slide_time=6110) [2024-04-22 16:46:27.567261] INFO [PALF] read (log_cache.cpp:100) [14085][T1_ReplaySrv6][T1][Y0-0000000000000000-0-0] [lt=201] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=845, hit_cnt=7, read_cnt=7, hit rate=1.000000000000000000e+00) [2024-04-22 16:46:27.567467] INFO [LIB] stat (utility.h:1165) [14086][T1_ReplaySrv7][T1][Y0-0000000000000000-0-0] [lt=7] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=76, this=0x7f4d98c048f8, extra_info="") [2024-04-22 16:46:27.567542] INFO [LIB] stat (utility.h:1165) [14086][T1_ReplaySrv7][T1][Y0-0000000000000000-0-0] [lt=72] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=10, this=0x7f4d98c04b28, extra_info="") [2024-04-22 16:46:27.568288] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=44][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:27.571425] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.571494] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=44000, remain_us=882620, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=44, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.573867] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.573959] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=14107970, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=201, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:27.575666] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=0) [2024-04-22 16:46:27.621191] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.621288] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=45000, remain_us=832827, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=45, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.630017] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:46:27.650157] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=72] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:27.670352] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.670464] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=46000, remain_us=783651, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=46, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.677797] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.677909] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=14004021, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=202, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:27.721094] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.721191] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=47000, remain_us=732923, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=47, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.730123] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=58][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:27.730198] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=70][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775587553945061, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775587730109) [2024-04-22 16:46:27.730291] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=89][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775587530243, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:27.732091] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC41-0-0] [lt=242][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:27.732148] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC41-0-0] [lt=53][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:27.732191] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC41-0-0] [lt=41][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:27.732221] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC41-0-0] [lt=28][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:27.771975] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.772068] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=48000, remain_us=682046, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=48, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.781448] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.781498] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=13900431, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=203, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:27.787342] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:705) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=50] Cache replace map node details(ret=0, replace_node_count=0, replace_time=11504, replace_start_pos=2642388, replace_num=62914) [2024-04-22 16:46:27.787425] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=78] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=10) [2024-04-22 16:46:27.825345] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.825434] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1] will sleep(sleep_us=49000, remain_us=628680, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=49, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.849009] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [13622][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=75][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:310, tid:13686}]) [2024-04-22 16:46:27.850892] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=71] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:27.874602] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:49, local_retry_times:49, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:27.874731] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=107][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.874754] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.874766] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:27.874843] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=50][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:27.879924] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=38][errcode=-4725] [2024-04-22 16:46:27.879970] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=43][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:27.880003] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:27.880022] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=17][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:27.880038] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:27.880050] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:27.880061] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.880108] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.880163] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=47][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.880185] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:27.880233] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=45][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=50, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.880274] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=24] will sleep(sleep_us=50000, remain_us=573841, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=50, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.881618] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=17][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:203, local_retry_times:203, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:27.881694] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.881710] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=14][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.881718] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=7][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:27.881747] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=7][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:27.886063] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] [2024-04-22 16:46:27.886092] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=27][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:27.886110] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:27.886123] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:27.886132] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:27.886139] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=6][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:27.886157] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=17][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.886184] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=6] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.886210] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=22][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.886219] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=7][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:27.886229] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=204, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.886247] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10] will sleep(sleep_us=100000, remain_us=13795681, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=204, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:27.930332] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:27.930416] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=80][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775587744021666, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775587930315) [2024-04-22 16:46:27.930440] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775587730317, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:27.930446] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:50, local_retry_times:50, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:27.930499] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=34][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.930531] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=28][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.930544] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:27.930569] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:27.934241] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] [2024-04-22 16:46:27.934276] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=33][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:27.934299] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:27.934316] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:27.934361] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=42][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:27.934375] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:27.934387] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.934421] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.934459] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=32][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.934475] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:27.934488] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=51, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.934513] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12] will sleep(sleep_us=51000, remain_us=519601, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=51, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.935663] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=35][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:27.935722] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=40] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:46:27.935747] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=22][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-04-22 16:46:27.935762] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=14][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-04-22 16:46:27.935776] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-04-22 16:46:27.935791] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=10][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:27.935801] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=9][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:27.935815] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=8][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-04-22 16:46:27.935824] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=7][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:27.935833] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=8][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:27.935841] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=7][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:27.935849] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:4307) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=7][errcode=-5019] resolve table failed(ret=-5019) [2024-04-22 16:46:27.935873] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:3867) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=22][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-04-22 16:46:27.935881] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3723) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=7][errcode=-5019] resolve joined table failed(ret=-5019) [2024-04-22 16:46:27.935890] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:27.935899] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=7][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:27.935907] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:27.935916] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:27.935941] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=15][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:27.935962] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=17][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:27.935975] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:27.935984] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=8][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:27.935993] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=7][errcode=-5019] fail to handle text query(stmt=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-04-22 16:46:27.936012] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=17][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:27.936022] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.936050] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=22][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:27.936092] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=35][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:27.936102] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:27.936110] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=7][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:27.936128] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14114][BlackListServic][T1][YB42AC175883-000616AB07E7349F-0-0] [lt=8][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:27.936142] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [14114][BlackListServic][T0][YB42AC175883-000616AB07E7349F-0-0] [lt=11][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-04-22 16:46:27.936154] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:46:27.936175] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:27.936185] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:27.936196] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7e008050, start=1713775587935306, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:46:27.936219] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:27.936236] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:219) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-04-22 16:46:27.936323] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:193) [14114][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=18] ls blacklist refresh finish(cost_time=2266) [2024-04-22 16:46:27.938579] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [14062][T1_LSArchiveMgr][T1][YB42AC175883-000616AB07773224-0-0] [lt=43] gc stale ls task succ [2024-04-22 16:46:27.940452] INFO do_work (ob_rl_mgr.cpp:704) [13683][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=39] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:46:27.940492] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [14156][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=38] current monitor number(seq_=-1) [2024-04-22 16:46:27.942082] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [14055][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=0, size_used=13370, mem_used=68631552) [2024-04-22 16:46:27.942982] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [14072][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=45] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:46:27.947315] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=5][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:27.947387] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=56] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, table_name.ptr()="data_size:15, data:5F5F616C6C5F6C735F737461747573", ret=-5019) [2024-04-22 16:46:27.947362] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] [COMPACTION DAG STATUS] (start_time=1713775577944040, end_time=1713775587947359, dag_status={}) [2024-04-22 16:46:27.947412] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=22][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, ret=-5019) [2024-04-22 16:46:27.947417] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=50] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:46:27.947428] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=14][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_status, db_name=oceanbase) [2024-04-22 16:46:27.947440] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:46:27.947443] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=14][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_status) [2024-04-22 16:46:27.947454] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:46:27.947458] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=11][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:27.947468] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=9][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:27.947466] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:46:27.947481] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=8][errcode=-5019] Table 'oceanbase.__all_ls_status' doesn't exist [2024-04-22 16:46:27.947478] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:46:27.947491] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=9][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:27.947491] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:46:27.947500] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:27.947503] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:46:27.947509] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=8][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:27.947518] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:27.947515] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:46:27.947527] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2818) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2024-04-22 16:46:27.947528] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:27.947545] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=15][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:27.947555] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:27.947540] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947572] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=9][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:27.947567] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=26] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947583] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:27.947582] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947595] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:27.947597] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947604] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=8][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:27.947613] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", ret=-5019) [2024-04-22 16:46:27.947612] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947654] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=37][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:27.947657] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=43] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947666] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.947673] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947687] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947687] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=16][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:27.947708] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:27.947702] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947717] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:27.947726] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947732] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=14][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:27.947741] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:0, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947755] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947763] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=9][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:27.947770] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947776] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [14027][T1_DupTbLease][T1][YB42AC175883-000616AB096732C2-0-0] [lt=12][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, aret=-5019, ret=-5019) [2024-04-22 16:46:27.947784] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947789] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [14027][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-04-22 16:46:27.947800] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [14027][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:27.947798] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947810] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [14027][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:27.947812] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947821] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [14027][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7c05c050, start=1713775587947163, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-04-22 16:46:27.947827] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947834] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [14027][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:27.947845] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:943) [14027][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] failed to read(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-04-22 16:46:27.947841] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947870] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=28] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947892] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947921] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:672) [14027][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] fail to inner get ls status info(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", tenant_id=1, exec_tenant_id=1, need_member_list=false) [2024-04-22 16:46:27.947928] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=35] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947944] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947939] WDIAG [STORAGE.DUP_TABLE] refresh_dup_tablet_schema_ (ob_dup_table_util.cpp:97) [14027][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] get duplicate ls status info failed(ret=-5019, dup_ls_status_info={tenant_id:0, ls_id:{id:-1}, ls_group_id:18446744073709551615, status:"UNKNOWN", unit_group_id:18446744073709551615, primary_zone:"", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2024-04-22 16:46:27.947959] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947973] WDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:148) [14027][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-5019] refresh dup table schema failed(ret=-5019) [2024-04-22 16:46:27.947973] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947984] WDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:236) [14027][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] scan dup ls to find dup_tablet failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:27.947987] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.947994] WDIAG [STORAGE.DUP_TABLE] runTimerTask (ob_dup_table_util.cpp:73) [14027][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute dup ls scan failed(tmp_ret=-5019) [2024-04-22 16:46:27.948001] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948022] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=20] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948036] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948050] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948064] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948078] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948093] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948106] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948130] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948162] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=31] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948186] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948206] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=19] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948226] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=20] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948294] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=66] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948317] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948341] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=24] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948364] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=22] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948388] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948420] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=31] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948435] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948449] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948464] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948477] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948492] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948516] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948538] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=21] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948571] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=32] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948591] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=19] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948605] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948619] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948663] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=43] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948678] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948692] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4252) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2024-04-22 16:46:27.948708] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3583) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2024-04-22 16:46:27.948720] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2024-04-22 16:46:27.948740] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=19] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2024-04-22 16:46:27.948750] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2024-04-22 16:46:27.948759] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2024-04-22 16:46:27.948768] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2024-04-22 16:46:27.948777] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2024-04-22 16:46:27.948785] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2024-04-22 16:46:27.948794] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2024-04-22 16:46:27.948802] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3586) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2024-04-22 16:46:27.948813] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4271) [14008][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(dag_cnt=0, total_worker_cnt=43, total_running_task_cnt=0, work_thread_num=43, scheduled_task_cnt=0) [2024-04-22 16:46:27.958476] INFO pn_ratelimit (group.c:589) [13817][IngressService][T0][Y0-0000000000000000-0-0] [lt=21] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-04-22 16:46:27.962385] WDIAG load_file_to_string (utility.h:627) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:46:27.962448] WDIAG get_ethernet_speed (utility.cpp:580) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:46:27.962471] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:46:27.962512] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:46:27.985618] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:51, local_retry_times:51, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:27.985677] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=48][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.985688] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.985699] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:27.985725] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:27.986385] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:204, local_retry_times:204, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:27.986471] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.986499] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=24][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:27.986512] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:27.986538] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:27.987617] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=9) [2024-04-22 16:46:27.989345] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] [2024-04-22 16:46:27.989361] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:27.989374] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:27.989392] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:27.989402] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:27.989410] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:27.989417] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=6][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.989438] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.989470] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=29][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.989476] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:27.989485] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=52, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.989502] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] will sleep(sleep_us=52000, remain_us=464612, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=52, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:27.990472] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=50][errcode=-4725] [2024-04-22 16:46:27.990533] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=58][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:27.990589] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=53][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:27.990604] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:27.990620] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=15][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:27.990708] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=85][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:27.990729] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=20][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.990786] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:27.990836] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=42][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:27.990857] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:27.990905] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=45][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=205, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:27.990953] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=31] will sleep(sleep_us=100000, remain_us=13690976, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=205, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:28.041591] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:52, local_retry_times:52, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:28.041650] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=48][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.041669] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.041675] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:28.041689] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=6][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:28.045009] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] [2024-04-22 16:46:28.045025] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:28.045037] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:28.045047] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:28.045055] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:28.045061] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:28.045067] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.045086] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=4] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.045104] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.045119] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:28.045128] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=53, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:28.045145] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] will sleep(sleep_us=53000, remain_us=408969, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=53, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:28.046887] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13806][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=66] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.047225] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13802][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.047378] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13787][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.047378] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13799][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=43] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.047477] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13779][RpcIO][T0][Y0-0000000000000000-0-0] [lt=47] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.047751] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13794][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.047856] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13797][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=44] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.048022] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13793][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.048108] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13801][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=46] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.048137] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13800][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.048348] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13780][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.048418] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13798][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=39] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.048484] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13783][RpcIO][T0][Y0-0000000000000000-0-0] [lt=48] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.048655] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13784][RpcIO][T0][Y0-0000000000000000-0-0] [lt=50] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.048668] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13796][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=26] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.048709] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13791][BatchIO][T0][Y0-0000000000000000-0-0] [lt=88] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.048762] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13782][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.048783] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13804][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=30] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.048835] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13785][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.048913] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13792][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.049023] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13788][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.049072] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13790][BatchIO][T0][Y0-0000000000000000-0-0] [lt=57] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.049152] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13789][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.049707] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13781][RpcIO][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:28.051715] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=66] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:28.069755] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=18][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:28.069791] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=23] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:46:28.069828] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=34][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:46:28.069839] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=10][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:46:28.069850] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=10][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:46:28.069861] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=7][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:28.069868] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=6][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:28.069877] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=5][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:46:28.069884] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=6][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:28.069891] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=7][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:28.069898] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=6][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:28.069904] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=6][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:28.070045] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=133][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:28.070081] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=34][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:28.070098] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=14][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:28.070134] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=17][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:28.070159] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=20][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:28.070181] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=16][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:28.070195] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:28.070208] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:46:28.070225] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=14][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:28.070240] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=13][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:28.070275] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=26][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:28.070331] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=46][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:28.070350] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:28.070367] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=16][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:28.070403] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397C-0-0] [lt=18][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:28.070432] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397C-0-0] [lt=24][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:46:28.070449] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397C-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:28.070465] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397C-0-0] [lt=14][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:28.070477] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397C-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:28.070491] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397C-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f4d9b5f2050, start=1713775588069581, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:28.070511] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397C-0-0] [lt=18][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:28.070525] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397C-0-0] [lt=10][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:46:28.070706] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397C-0-0] [lt=28][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:46:28.070775] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397C-0-0] [lt=66][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:46:28.091105] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=28][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:205, local_retry_times:205, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:28.091191] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.091219] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.091255] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=34][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:28.091291] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:28.094962] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] [2024-04-22 16:46:28.095000] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=35][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:28.095029] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:28.095046] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:28.095060] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:28.095071] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:28.095083] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.095129] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.095187] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=50][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.095208] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:28.095269] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=57][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=206, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:28.095308] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=24] will sleep(sleep_us=100000, remain_us=13586622, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=206, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:28.098230] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:53, local_retry_times:53, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:28.098261] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=23][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.098278] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.098286] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:28.098300] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:28.101749] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] [2024-04-22 16:46:28.101765] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:28.101776] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:28.101785] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:28.101794] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:28.101801] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:28.101807] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=6][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.101825] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.101840] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.101852] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:28.101861] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=54, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:28.101874] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8] will sleep(sleep_us=54000, remain_us=352239, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=54, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:28.130238] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=53] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:46:28.130541] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:28.130703] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=76][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775587946966197, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775588130523) [2024-04-22 16:46:28.130750] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775587930470, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:28.155963] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:54, local_retry_times:54, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:28.156004] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=30][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.156014] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.156022] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:28.156047] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:28.159373] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] [2024-04-22 16:46:28.159390] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:28.159409] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:28.159418] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=8][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:28.159428] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:28.159436] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:28.159444] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.159465] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=6] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.159482] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.159488] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=5][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:28.159496] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=7][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=55, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:28.159512] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10] will sleep(sleep_us=55000, remain_us=294601, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=55, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:28.161549] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7D4-0-0] [lt=26] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=0, this=0x7f4d7e495470, extra_info="") [2024-04-22 16:46:28.161701] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7D4-0-0] [lt=59] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.131:2882", role="FOLLOWER", total_group_log_cnt=5, avg_log_batch_cnt=0, total_group_log_size=610, avg_group_log_size=122) [2024-04-22 16:46:28.161738] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7D4-0-0] [lt=33] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.131:2882", committed size=610) [2024-04-22 16:46:28.167170] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:783) [13729][AutoLSLocRpc][T0][YB42AC175883-000616AB09B7392D-0-0] [lt=95] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1713772869}, location:{server:"172.23.88.129:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:18}}]) [2024-04-22 16:46:28.186401] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [13693][LSMetaTblUp0][T0][YB42AC175883-000616AB08B9F936-0-0] [lt=67][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713775570633812}) [2024-04-22 16:46:28.187856] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=90] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=8) [2024-04-22 16:46:28.195465] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:206, local_retry_times:206, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:28.195569] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=78][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.195592] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.195604] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:28.195664] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:28.199590] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=39][errcode=-4725] [2024-04-22 16:46:28.199684] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=91][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:28.199736] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=49][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:28.199751] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:28.199766] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:28.199778] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:28.199790] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.199838] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.199897] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=50][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.199927] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=26][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:28.199957] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=28][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=207, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:28.200003] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=27] will sleep(sleep_us=100000, remain_us=13481927, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=207, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:28.214648] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:55, local_retry_times:55, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:28.214742] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=75][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.214768] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=22][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.214780] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:28.214804] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:28.215823] INFO eloop_run (eloop.c:140) [13667][pnio1][T0][Y0-0000000000000000-0-0] [lt=104] PNIO [ratelimit] time: 1713775588215819, bytes: 91702250, bw: 0.018308 MB/s, add_ts: 1007618, add_bytes: 19344 [2024-04-22 16:46:28.217755] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=17][errcode=-4725] [2024-04-22 16:46:28.217804] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=47][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:28.217841] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=34][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:28.217858] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:28.217870] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:28.217880] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:28.217890] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.217934] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.217990] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=49][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.218003] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:28.218019] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=56, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:28.218049] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16] will sleep(sleep_us=56000, remain_us=236066, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=56, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:28.223665] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=10] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:18, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713775573499769, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"]) [2024-04-22 16:46:28.225940] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=165][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:28.226018] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=57] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:46:28.226058] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=34][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:46:28.226082] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=22][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:46:28.226102] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:46:28.226122] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:28.226135] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:28.226371] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=216][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:46:28.226400] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=25][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:28.226419] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=17][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:28.226432] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=11][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:28.226444] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:28.226456] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:28.226470] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=12][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:28.226484] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:28.226526] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=19][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:28.226577] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=42][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:28.226609] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=24][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:28.226624] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:28.226755] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=123][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:46:28.226793] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=33][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:28.226809] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=14][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:28.226838] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=23][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:28.226875] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=30][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:28.226891] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=15][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:28.226906] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=14][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:28.226940] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=19][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:28.226960] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=18][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:46:28.226985] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=20][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:28.227017] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=28][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:28.227078] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=57][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:28.227100] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=19][errcode=-5019] query failed(ret=-5019, conn=0x7f4dbdf7a050, start=1713775588225611, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:28.227121] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=19][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:28.227136] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:28.227263] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=15][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:46:28.227287] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=21][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:46:28.227318] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=29][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:28.227338] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=18][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:28.227361] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=16][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:46:28.227386] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=19][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:46:28.227400] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173934-0-0] [lt=14][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:28.230193] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:123) [13953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=71] ====== tenant freeze timer task ====== [2024-04-22 16:46:28.238282] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC44-0-0] [lt=263][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:28.238347] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC44-0-0] [lt=62][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:28.238397] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC44-0-0] [lt=47][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:28.238441] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC44-0-0] [lt=41][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:28.239068] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.239115] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.239136] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.239150] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.239162] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.239173] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.239184] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.239195] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.239205] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.239233] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.239241] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.239249] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.239256] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.239263] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.239270] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.239276] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:28.252408] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=64] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20814036992, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:28.269529] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [14065][T1_ArcSender][T1][YB42AC175883-000616AB07A73224-0-0] [lt=55][errcode=-4018] try free send task failed(ret=-4018) [2024-04-22 16:46:28.274216] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=17][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:56, local_retry_times:56, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:28.274339] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=81][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.274369] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=27][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.274386] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:28.274448] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:28.278209] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=51][errcode=-4725] [2024-04-22 16:46:28.278250] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=38][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:28.278300] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=46][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:28.278318] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:28.278333] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=14][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:28.278345] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:28.278356] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.278409] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.278450] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=35][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.278463] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:28.278481] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=17][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=57, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:28.278516] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=19] will sleep(sleep_us=57000, remain_us=175598, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=57, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:28.287385] INFO eloop_run (eloop.c:140) [13674][pnio2][T0][Y0-0000000000000000-0-0] [lt=55] PNIO [ratelimit] time: 1713775588287381, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007638, add_bytes: 0 [2024-04-22 16:46:28.300233] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:207, local_retry_times:207, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:28.300307] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=55][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.300348] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=38][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.300360] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:28.300381] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:28.304169] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=15][errcode=-4725] [2024-04-22 16:46:28.304194] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=23][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:28.304211] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:28.304230] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:28.304243] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:28.304250] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=6][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:28.304256] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=5][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.304282] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=5] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.304304] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=18][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.304311] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=5][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:28.304332] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=208, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:28.304352] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10] will sleep(sleep_us=100000, remain_us=13377577, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=208, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:28.330763] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:46:28.330799] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:28.330826] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775588159556911, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775588330753) [2024-04-22 16:46:28.330843] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775588130777, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:28.335648] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:57, local_retry_times:57, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:28.335694] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=33][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.335708] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=12][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.335718] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:28.335736] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=10][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:28.339131] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.339178] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=0] will sleep(sleep_us=58000, remain_us=114936, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=58, v.err_=-4725, timeout_timestamp=1713775588454113) [2024-04-22 16:46:28.369754] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:262) [14065][T1_ArcSender][T1][YB42AC175883-000616AB07A73224-0-0] [lt=61] ObArchiveSender is running(thread_index=0) [2024-04-22 16:46:28.388028] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=63] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=7) [2024-04-22 16:46:28.401438] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=1][errcode=-4012] [2024-04-22 16:46:28.401489] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=48][errcode=-4012] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4012, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:28.401524] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=31][errcode=-4012] failed to setup first scanner(ret=-4012) [2024-04-22 16:46:28.401559] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=30][errcode=-4012] Open this operator failed(ret=-4012, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:28.401682] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=50][errcode=-4012] open operator failed(ret=-4012) [2024-04-22 16:46:28.401708] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=24][errcode=-4012] fail open main query(ret=-4012) [2024-04-22 16:46:28.401725] WDIAG [SQL] open (ob_result_set.cpp:160) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:46:28.401744] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4012] open result set failed(ret=-4012) [2024-04-22 16:46:28.401762] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:28.401791] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=27][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=59, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:28.401815] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:59, local_retry_times:59, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-04-22 16:46:28.401850] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=24][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:46:28.401862] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4012] result set close failed(ret=-4012) [2024-04-22 16:46:28.401872] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=9][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-04-22 16:46:28.401894] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13686][SerScheQueue1][T1][YB42AC175883-000616AB09773315-0-0] [lt=11][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-04-22 16:46:28.401914] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:665) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=16] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1946244) [2024-04-22 16:46:28.401967] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=50][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-04-22 16:46:28.402003] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=22][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:46:28.402044] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=38][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-04-22 16:46:28.402067] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=20][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:28.402085] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=15][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-04-22 16:46:28.402109] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-04-22 16:46:28.402214] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=16][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-04-22 16:46:28.402256] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=39][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-04-22 16:46:28.402279] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=21][errcode=-4012] get failed(ret=-4012) [2024-04-22 16:46:28.402298] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:888) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=17][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:46:28.402359] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4507) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=57][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-04-22 16:46:28.402379] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:521) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=18][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-04-22 16:46:28.402395] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:236) [13686][SerScheQueue1][T0][YB42AC175883-000616AB09773315-0-0] [lt=13][errcode=-4012] fail to process refresh task(ret=-4012, ret="OB_TIMEOUT", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-04-22 16:46:28.402421] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:599) [13686][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4012] fail to batch process task(ret=-4012) [2024-04-22 16:46:28.402433] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:499) [13686][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4012] fail to batch execute task(ret=-4012, tasks.count()=1) [2024-04-22 16:46:28.402501] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:261) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330F-0-0] [lt=64] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:46:28.402528] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:305) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330F-0-0] [lt=23] [REFRESH_SCHEMA] begin refresh schema, (begin_time=1713775588402527) [2024-04-22 16:46:28.402537] INFO [SHARE.SCHEMA] refresh_and_add_schema (ob_multi_version_schema_service.cpp:2388) [13685][SerScheQueue0][T0][YB42AC175883-000616AB09D7330F-0-0] [lt=8] [REFRESH_SCHEMA] start to refresh and add schema(tenant_ids=[]) [2024-04-22 16:46:28.404480] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:28.406002] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=15][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:28.406034] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.406060] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] already timeout, do not need sleep(sleep_us=0, remain_us=1996501, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.406102] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [13720][SysLocAsyncUp0][T0][YB42AC175883-000616AB07685540-0-0] [lt=49][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713775588406022}) [2024-04-22 16:46:28.407554] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.407594] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=13274335, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=209, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:28.409348] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.409376] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] will sleep(sleep_us=1000, remain_us=1993185, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.413651] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.413676] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] will sleep(sleep_us=2000, remain_us=1988885, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.419025] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.419057] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] will sleep(sleep_us=3000, remain_us=1983504, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.425582] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.425666] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] will sleep(sleep_us=4000, remain_us=1976896, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.433471] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.433529] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] will sleep(sleep_us=5000, remain_us=1969033, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.442244] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.442307] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=6000, remain_us=1960254, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.452016] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.452106] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] will sleep(sleep_us=7000, remain_us=1950456, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.453130] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=89] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:28.463120] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.463212] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=8000, remain_us=1939350, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.475309] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.475394] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] will sleep(sleep_us=9000, remain_us=1927168, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.488040] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.488090] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=10000, remain_us=1914472, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.501920] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.502007] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] will sleep(sleep_us=11000, remain_us=1900556, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.511768] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.511878] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=13170052, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=210, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:28.516709] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.516810] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=12000, remain_us=1885753, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.530959] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:28.531070] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=106][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775588361068024, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775588530940) [2024-04-22 16:46:28.531109] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775588330857, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:28.533719] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.533811] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] will sleep(sleep_us=13000, remain_us=1868751, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.550788] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.550876] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=14000, remain_us=1851686, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.557608] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [13778][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:46:28.565924] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=33][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:28.565993] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=57] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:28.566030] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=23] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:46:28.566055] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=21][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:46:28.566074] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:46:28.566093] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:46:28.566109] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=11] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:28.566121] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=10][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:28.566134] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:28.566152] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:46:28.566165] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:28.566199] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=33][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:28.566211] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=11][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:28.566223] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:28.566242] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=18][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:28.566260] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=15][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:28.566278] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=16][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:28.566329] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=37][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:28.566380] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=43][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:28.566407] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=22][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:28.566427] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=18][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:28.566440] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=11][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-04-22 16:46:28.566466] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=13][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:28.566535] WDIAG [SERVER] do_process (obmp_query.cpp:760) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=65][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-04-22 16:46:28.566572] WDIAG [SERVER] do_process (obmp_query.cpp:884) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=28][errcode=-5019] query failed(ret=-5019, session={this:0x7f4d7edd01f8, id:3221225477, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-04-22 16:46:28.566681] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352A-0-0] [lt=38] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0x579c808 0x578fbfb 0x5788b5f 0x577a048 0xabd78a4 0x16c398be 0x7f4dc8ea7ea5 0x7f4dc8bd0b0d") [2024-04-22 16:46:28.568791] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.568902] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] will sleep(sleep_us=15000, remain_us=1833661, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.581878] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:580) [14021][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=132] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:3, free_obj_hold_cnt:3, allocator used:23856, allocator total:65408}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=65408, tablets_mem_limit=858993440, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):640, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, pending_cnt=0, wait_gc_count=0, tablet count=3) [2024-04-22 16:46:28.587807] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.587892] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] will sleep(sleep_us=16000, remain_us=1814671, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.588182] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=6) [2024-04-22 16:46:28.607817] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.607907] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] will sleep(sleep_us=17000, remain_us=1794655, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.615656] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.615726] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=13066204, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=211, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:28.628531] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.628708] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=18000, remain_us=1773854, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.629872] INFO [CLOG] run1 (ob_garbage_collector.cpp:1348) [14037][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=41] Garbage Collector is running(seq_=191, gc_interval=10000000) [2024-04-22 16:46:28.631060] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-04-22 16:46:28.631172] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:273) [14037][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=47] GC check ls in member list, skip it(id={id:1}, leader="172.23.88.129:2882") [2024-04-22 16:46:28.631244] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1441) [14037][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=56] gc_check_member_list_ cost time(ret=0, time_us=1323) [2024-04-22 16:46:28.631292] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1713) [14037][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=34] execute_gc cost time(ret=0, time_us=1) [2024-04-22 16:46:28.631337] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1636) [14037][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=29] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2024-04-22 16:46:28.631373] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1658) [14037][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=25] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2024-04-22 16:46:28.631400] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1713) [14037][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=24] execute_gc cost time(ret=0, time_us=30) [2024-04-22 16:46:28.631421] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [14037][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=17] ObSafeDestroyHandler start process [2024-04-22 16:46:28.631438] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [14037][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=15] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2024-04-22 16:46:28.631469] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [14037][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=12] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2024-04-22 16:46:28.649995] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:189) [14116][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-04-22 16:46:28.650098] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:445) [14116][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=63] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-04-22 16:46:28.650150] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [14116][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=50] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-04-22 16:46:28.650376] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.650404] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] will sleep(sleep_us=19000, remain_us=1752158, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.653707] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=69] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:28.666265] INFO [LIB] stat (utility.h:1165) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT WRITE LOG COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=4587, this=0x7f4d7e49f330, extra_info="palf_id:1") [2024-04-22 16:46:28.666343] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2164) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=74] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"172.23.88.131:2882", has_set_deleted:false}, accum_size=732) [2024-04-22 16:46:28.666417] INFO [LIB] stat (utility.h:1182) [13928][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=59, this=0x7f4d95343700, extra_info="") [2024-04-22 16:46:28.666679] INFO [LIB] stat (utility.h:1165) [13927][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=53] [PALF STAT FLUSH CB COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=110, this=0x7f4d7e49f560, extra_info="palf_id:1") [2024-04-22 16:46:28.667800] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E7C-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=33, this=0x7f4d7e4956a0, extra_info="palf_id:1") [2024-04-22 16:46:28.667858] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E7C-0-0] [lt=54] [PALF STAT LOG LIFE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=5908, this=0x7f4d7e4958d0, extra_info="palf_id:1") [2024-04-22 16:46:28.667885] INFO [PALF] sliding_cb (log_sliding_window.cpp:2260) [17893][T1_L0_G1][T1][YB42AC175883-000616AB09175E7C-0-0] [lt=25] [PALF STAT LOG TASK TIME](palf_id=1, self="172.23.88.131:2882", total_slide_log_cnt=6, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=33, avg_log_submit_to_flush_time=4744, avg_log_submit_to_slide_time=5875) [2024-04-22 16:46:28.667868] INFO [PALF] read (log_cache.cpp:100) [14081][T1_ReplaySrv2][T1][Y0-0000000000000000-0-0] [lt=178] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=732, hit_cnt=6, read_cnt=6, hit rate=1.000000000000000000e+00) [2024-04-22 16:46:28.668080] INFO [LIB] stat (utility.h:1165) [14082][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=25] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=105, this=0x7f4d98c048f8, extra_info="") [2024-04-22 16:46:28.668170] INFO [LIB] stat (utility.h:1165) [14082][T1_ReplaySrv3][T1][Y0-0000000000000000-0-0] [lt=87] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=6, stat_interval=1000000, avg (count/cost)=11, this=0x7f4d98c04b28, extra_info="") [2024-04-22 16:46:28.672943] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.672996] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=20000, remain_us=1729566, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.673061] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [13930][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=51] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4096, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):65, sequence:2}) [2024-04-22 16:46:28.697265] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.697350] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=21000, remain_us=1705212, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.720010] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.720080] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=12961849, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=212, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:28.721961] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.722046] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] will sleep(sleep_us=22000, remain_us=1680516, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.731120] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=77][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:28.731175] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775588461811945, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775588731111) [2024-04-22 16:46:28.731191] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775588531135, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:28.733502] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC47-0-0] [lt=319][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:28.733561] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC47-0-0] [lt=56][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:28.733668] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC47-0-0] [lt=47][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:28.733708] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC47-0-0] [lt=38][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:28.747678] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.747798] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=23000, remain_us=1654764, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.773793] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.773879] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=24000, remain_us=1628684, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.788373] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=58] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=5) [2024-04-22 16:46:28.803104] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.803180] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=25000, remain_us=1599382, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.825352] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.825438] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=12856491, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=213, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:28.831715] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.831800] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=26000, remain_us=1570762, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.854466] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:28.861421] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.861507] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] will sleep(sleep_us=27000, remain_us=1541055, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.870670] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:648) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=89] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=16110, clean_start_pos=1384119, clean_num=125829) [2024-04-22 16:46:28.893060] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.893147] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=28000, remain_us=1509415, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.908030] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1918) [13951][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=16] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-04-22 16:46:28.908118] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1459) [13951][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=78] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=214748360, cache_obj_num=3, cache_node_num=3) [2024-04-22 16:46:28.908165] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1476) [13951][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=41] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=214748360, cache_obj_num=3, cache_node_num=3) [2024-04-22 16:46:28.908189] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2682) [13951][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=21] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:46:28.916818] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2401) [13951][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=37] Dumping All Cache Objs(alloc_obj_list.count()=3, alloc_obj_list=[{obj_id:5, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:156285}, {obj_id:99, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:175513}, {obj_id:100, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:176531}]) [2024-04-22 16:46:28.916866] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2690) [13951][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=43] schedule next cache evict task(evict_interval=5000000) [2024-04-22 16:46:28.925057] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.925154] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=29000, remain_us=1477408, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=29, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.929233] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.929318] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=12752611, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=214, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:28.931265] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:28.931298] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775588752831841, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775588931257) [2024-04-22 16:46:28.931323] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775588731203, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:28.938672] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:584) [14062][T1_LSArchiveMgr][T1][YB42AC175883-000616AB07773224-0-0] [lt=38] gc stale ls task succ [2024-04-22 16:46:28.940604] INFO do_work (ob_rl_mgr.cpp:704) [13683][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=40] swc wakeup.(stat_period_=1000000, ready=false) [2024-04-22 16:46:28.940658] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [14156][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=27] current monitor number(seq_=-1) [2024-04-22 16:46:28.942339] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [14055][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=61] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=103079214, evict_high_size_level=9000000, evict_batch_count=0, elapse_time=1, size_used=13422, mem_used=68631552) [2024-04-22 16:46:28.943150] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:188) [14072][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=50] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-04-22 16:46:28.952052] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [13622][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=47][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4725, dropped:550, tid:13686}]) [2024-04-22 16:46:28.954346] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=36][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:29, local_retry_times:29, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:28.954429] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=61][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.954455] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=23][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.954468] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=12][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:28.954493] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:28.958223] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=18][errcode=-4725] [2024-04-22 16:46:28.958270] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=44][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:28.958300] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:28.958322] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=18][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:28.958343] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=19][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:28.958356] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:28.958366] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=9][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.958410] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=8] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.958447] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=30][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.958459] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:28.958475] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=30, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:28.958506] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=17] will sleep(sleep_us=30000, remain_us=1444056, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=30, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:28.963010] WDIAG load_file_to_string (utility.h:627) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] read /sys/class/net/eth0/speed failed, errno 22 [2024-04-22 16:46:28.963094] WDIAG get_ethernet_speed (utility.cpp:580) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=64][errcode=-4000] load file /sys/class/net/eth0/speed failed, ret -4000 [2024-04-22 16:46:28.963126] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:2768) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="eth0") [2024-04-22 16:46:28.963153] WDIAG [SERVER] runTimerTask (ob_server.cpp:3290) [13614][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2024-04-22 16:46:28.988585] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=70] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=4) [2024-04-22 16:46:28.988761] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=18][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:30, local_retry_times:30, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:28.988848] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=65][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.988870] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:28.988881] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:28.988930] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:28.992495] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=21][errcode=-4725] [2024-04-22 16:46:28.992543] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=46][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:28.992576] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:28.992595] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:28.992609] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:28.992620] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:28.992672] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=50][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.992720] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:28.992760] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=33][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:28.992773] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:28.992791] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=31, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:28.992824] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=19] will sleep(sleep_us=31000, remain_us=1409738, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=31, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:29.024064] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:31, local_retry_times:31, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:29.024152] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=67][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.024174] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.024185] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:29.024211] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:29.027862] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=19][errcode=-4725] [2024-04-22 16:46:29.027903] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=38][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:29.027933] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=27][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:29.027949] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:29.027963] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:29.027975] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=11][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:29.027986] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.028031] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.028087] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=48][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.028109] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=19][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:29.028130] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=32, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:29.028172] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=19] will sleep(sleep_us=32000, remain_us=1374390, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=32, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:29.029548] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=25][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:214, local_retry_times:214, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:29.029721] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=153][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.029746] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=22][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.029758] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:29.029779] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:29.031393] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:29.031486] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=87][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775588861567829, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775589031374) [2024-04-22 16:46:29.031564] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=74][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775588931334, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:29.033823] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] [2024-04-22 16:46:29.033847] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=23][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:29.033864] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=15][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:29.033877] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:29.033888] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:29.033895] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=7][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:29.033901] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=6][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.033926] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=5] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.033959] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=28][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.033965] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=5][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:29.033975] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=215, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:29.033993] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10] will sleep(sleep_us=100000, remain_us=12647936, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=215, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:29.047343] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13787][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.047373] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13802][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=60] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.047383] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13794][BatchIO][T0][Y0-0000000000000000-0-0] [lt=38] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.047565] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13783][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.047577] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13779][RpcIO][T0][Y0-0000000000000000-0-0] [lt=40] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.047719] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13799][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=42] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.047736] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13793][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.048015] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13806][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=46] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.048000] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13797][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=28] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.048242] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13796][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=37] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.048258] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13800][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=33] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.048264] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13785][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.048286] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13801][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=42] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.048331] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13790][BatchIO][T0][Y0-0000000000000000-0-0] [lt=46] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.048341] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13792][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.048436] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13780][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.048448] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13798][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.048817] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13791][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.048838] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [13804][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=29] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.049251] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13788][BatchIO][T0][Y0-0000000000000000-0-0] [lt=37] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.049320] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13782][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.049737] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13781][RpcIO][T0][Y0-0000000000000000-0-0] [lt=43] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.049793] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [13784][RpcIO][T0][Y0-0000000000000000-0-0] [lt=42] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.050272] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [13789][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-04-22 16:46:29.060301] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:32, local_retry_times:32, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:29.060385] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.060446] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=58][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.060458] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:29.060505] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:29.064279] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=31][errcode=-4725] [2024-04-22 16:46:29.064326] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=44][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:29.064358] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:29.064377] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:29.064399] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:29.064416] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=15][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:29.064435] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=17][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.064500] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=13] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.064573] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=63][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.064596] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=20][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:29.064668] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=24][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=33, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:29.064732] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=37] will sleep(sleep_us=33000, remain_us=1337831, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=33, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:29.071294] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:29.072933] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=18][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:29.073011] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=56] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-04-22 16:46:29.073044] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-04-22 16:46:29.073065] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=19][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-04-22 16:46:29.073085] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-04-22 16:46:29.073104] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:29.073117] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:29.073134] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-04-22 16:46:29.073147] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:29.073159] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:29.073170] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=9][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:29.073180] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:29.073192] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:29.073204] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:29.073216] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:29.073241] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:29.073255] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:29.073270] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:29.073282] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:29.073306] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=21][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-04-22 16:46:29.073321] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:29.073333] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:29.073360] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:29.073386] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:29.073397] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:29.073407] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:29.073429] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13620][ConfigMgr][T1][YB42AC175883-000616AB0997397D-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:29.073447] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397D-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-04-22 16:46:29.073463] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397D-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:29.073477] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397D-0-0] [lt=12][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:29.073489] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397D-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:29.073503] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397D-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f4da0e3e050, start=1713775589072681, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-04-22 16:46:29.073520] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397D-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:29.073533] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397D-0-0] [lt=10][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-04-22 16:46:29.073707] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397D-0-0] [lt=13][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-04-22 16:46:29.073725] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [13620][ConfigMgr][T0][YB42AC175883-000616AB0997397D-0-0] [lt=17][errcode=-5019] Update local config failed(ret=-5019) [2024-04-22 16:46:29.097872] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=27][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:33, local_retry_times:33, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:29.097953] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=60][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.097981] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.098000] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=17][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:29.098055] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=19][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:29.101346] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=23][errcode=-4725] [2024-04-22 16:46:29.101392] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=43][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:29.101424] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:29.101449] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=21][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:29.101471] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=20][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:29.101491] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=18][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:29.101510] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=17][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.101559] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=11] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.101610] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=43][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.101694] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=81][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:29.101743] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=34, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:29.101783] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=24] will sleep(sleep_us=34000, remain_us=1300779, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=34, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:29.131226] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [13921][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=69] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-04-22 16:46:29.131517] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:540) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1713775588861567829, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=269943, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1713775589131510, force_self_check=false, weak_read_refresh_interval=100000) [2024-04-22 16:46:29.131679] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [13932][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=95] LogLoopThread round_cost_time(us)(round_cost_time=38) [2024-04-22 16:46:29.134124] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:215, local_retry_times:215, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:29.134212] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.134242] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=27][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.134259] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:29.134323] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=18][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:29.135934] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=24][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:34, local_retry_times:34, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:29.136013] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=60][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.136041] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.136061] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=18][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:29.136089] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=15][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:29.137903] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=39][errcode=-4725] [2024-04-22 16:46:29.137946] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:29.137976] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:29.137993] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:29.138006] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=12][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:29.138018] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:29.138048] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=28][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.138090] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.138137] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=40][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.138150] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:29.138167] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=15][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=216, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:29.138200] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=18] will sleep(sleep_us=100000, remain_us=12543730, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=216, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:29.139816] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=18][errcode=-4725] [2024-04-22 16:46:29.139857] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=38][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:29.139886] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:29.139903] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:29.139917] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:29.139928] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:29.139939] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.139984] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.140038] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=46][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.140059] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:29.140079] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=35, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:29.140121] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=19] will sleep(sleep_us=35000, remain_us=1262442, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=35, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:29.140742] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:349) [14059][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=24] ====== check clog disk timer task ====== [2024-04-22 16:46:29.140823] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [14059][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=70] get_disk_usage(ret=0, capacity(MB):=4096, used(MB):=65) [2024-04-22 16:46:29.140868] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:261) [14059][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=22] cannot_recycle_log_size statistics(cannot_recycle_log_size=1379748, threshold=1288490188, need_update_checkpoint_scn=false) [2024-04-22 16:46:29.162705] INFO [LIB] stat (utility.h:1165) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7E2-0-0] [lt=30] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=7, stat_interval=1000000, avg (count/cost)=0, this=0x7f4d7e495470, extra_info="") [2024-04-22 16:46:29.162819] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1090) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7E2-0-0] [lt=73] [PALF STAT GROUP LOG INFO](palf_id=1, self="172.23.88.131:2882", role="FOLLOWER", total_group_log_cnt=7, avg_log_batch_cnt=0, total_group_log_size=854, avg_group_log_size=122) [2024-04-22 16:46:29.162864] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1747) [17893][T1_L0_G1][T1][YB42AC175881-000616AB0F8FF7E2-0-0] [lt=40] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="172.23.88.131:2882", committed size=854) [2024-04-22 16:46:29.175269] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:35, local_retry_times:35, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:29.175353] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.175375] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.175386] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:29.175433] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:29.177822] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=29][errcode=-4725] [2024-04-22 16:46:29.177864] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=40][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:29.177893] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=26][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:29.177909] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=14][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:29.177923] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:29.177934] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:29.177945] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.178002] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=12] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.178073] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=62][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.178095] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=13][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:29.178115] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=18][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=36, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:29.178147] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=18] will sleep(sleep_us=36000, remain_us=1224415, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=36, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:29.187749] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:325) [13692][LSSysTblUp0][T0][YB42AC175883-000616AB08A9F928-0-0] [lt=29][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1713775570633812}) [2024-04-22 16:46:29.188853] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=114] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=3) [2024-04-22 16:46:29.214314] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=36][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:36, local_retry_times:36, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:29.214397] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=63][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.214425] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=25][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.214439] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=13][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:29.214464] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=12][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:29.218094] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=19][errcode=-4725] [2024-04-22 16:46:29.218143] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=46][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:29.218176] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=30][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:29.218195] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=16][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:29.218209] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:29.218221] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:29.218231] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.218277] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.218338] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=52][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.218360] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:29.218380] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=37, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:29.218412] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=19] will sleep(sleep_us=37000, remain_us=1184150, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=37, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:29.223469] INFO eloop_run (eloop.c:140) [13667][pnio1][T0][Y0-0000000000000000-0-0] [lt=83] PNIO [ratelimit] time: 1713775589223465, bytes: 91730411, bw: 0.026653 MB/s, add_ts: 1007646, add_bytes: 28161 [2024-04-22 16:46:29.228912] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:142) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=33] fetch root ls success(local_cluster_id=1713772869, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.129:2882", sql_port:2881, role:1, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:18, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, {modify_time_us:1713775573499769, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.23.88.131:2882", sql_port:2881, role:2, member_list:[{server:"172.23.88.129:2882", timestamp:1}, {server:"172.23.88.130:2882", timestamp:1}, {server:"172.23.88.131:2882", timestamp:1}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}, rs_list=["172.23.88.129:2882", "172.23.88.131:2882", "172.23.88.130:2882"]) [2024-04-22 16:46:29.231065] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [14073][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=210] tx gc loop thread is running(MTL_ID()=1) [2024-04-22 16:46:29.231119] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [14073][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=50] try gc retain ctx [2024-04-22 16:46:29.231539] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=149][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:29.231611] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=168][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:29.231621] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=60] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-04-22 16:46:29.231694] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=68][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-04-22 16:46:29.231707] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=66][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775589049509113, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775589231600) [2024-04-22 16:46:29.231714] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-04-22 16:46:29.231737] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775589031603, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:29.231755] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=38][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-04-22 16:46:29.231777] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=14][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:29.231790] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:29.231817] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-04-22 16:46:29.231832] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=12][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:29.231843] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:29.231854] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=10][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:29.231865] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:29.231877] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:29.231889] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:29.231900] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:29.231942] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=29][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:29.231956] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:29.231971] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:29.231983] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:29.231994] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-04-22 16:46:29.232010] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:686) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2024-04-22 16:46:29.232022] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:29.232060] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=31][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:29.232087] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:29.232098] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2024-04-22 16:46:29.232108] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-04-22 16:46:29.232151] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=21][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-04-22 16:46:29.232167] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:895) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-04-22 16:46:29.232182] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1682) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:29.232205] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:952) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=21][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-04-22 16:46:29.232219] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1622) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=12][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1713772869, tenant_id=1) [2024-04-22 16:46:29.232232] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=11][errcode=-5019] query failed(ret=-5019, conn=0x7f4d7c0fc050, start=1713775589231308, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:29.232250] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=17][errcode=-5019] read failed(ret=-5019) [2024-04-22 16:46:29.232264] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-04-22 16:46:29.232361] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=14][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-04-22 16:46:29.232386] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=22][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-04-22 16:46:29.232402] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=15][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:29.232434] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=30][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:29.232451] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=13][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:46:29.232468] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=14][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-04-22 16:46:29.232479] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [14043][T1_LSMetaCh][T1][YB42AC175883-000616AB08173935-0-0] [lt=11][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-04-22 16:46:29.234021] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:253) [14073][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=47] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1713773717210958135, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:0, v:0}, all_id_meta:{id_meta:[{limited_id:1713775597210958129, latest_log_ts:{val:1713775580545744419, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:20000001, latest_log_ts:{val:1713773717210958135, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x7f4d953f4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2024-04-22 16:46:29.237356] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC4A-0-0] [lt=372][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-04-22 16:46:29.237398] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC4A-0-0] [lt=40][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:29.237426] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC4A-0-0] [lt=26][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-04-22 16:46:29.237445] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [14133][T1_L0_G2][T1][YB42AC175881-000616AB08AFFC4A-0-0] [lt=18][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-04-22 16:46:29.238376] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=19][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:216, local_retry_times:216, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:29.238457] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=64][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.238474] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.238484] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:29.238504] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:29.240673] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=27][errcode=-4725] [2024-04-22 16:46:29.240712] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=37][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:29.240750] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=36][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:29.240763] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=11][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:29.240774] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=10][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:29.240783] WDIAG [SQL] open_result (ob_result_set.cpp:211) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=8][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:29.240800] WDIAG [SQL] open (ob_result_set.cpp:160) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=16][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.240840] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=7] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.240871] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=26][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.240882] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}) [2024-04-22 16:46:29.240896] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=13][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=217, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:29.240878] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.240913] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.240923] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=15] will sleep(sleep_us=100000, remain_us=12441006, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=217, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:29.240956] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS SENDER](send_index=3, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.240980] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=4, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.241002] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=5, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.241017] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](send_index=6, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.241040] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=7, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.241059] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](send_index=8, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.241076] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS SENDER](send_index=9, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.241092] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=10, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.241112] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](send_index=11, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.241174] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=58] [IO STATUS SENDER](send_index=12, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.241198] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS SENDER](send_index=13, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.241216] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS SENDER](send_index=14, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.241246] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS SENDER](send_index=15, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.241291] INFO [COMMON] print_sender_status (ob_io_struct.cpp:855) [13623][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=42] [IO STATUS SENDER](send_index=16, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-04-22 16:46:29.255544] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:37, local_retry_times:37, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:29.255617] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=55][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.255671] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=51][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.255682] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:29.255705] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=11][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:29.258347] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=30][errcode=-4725] [2024-04-22 16:46:29.258397] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=47][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:29.258430] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=29][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:29.258448] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=15][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:29.258462] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:29.258473] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:29.258484] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.258530] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.258586] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=47][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.258608] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=18][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:29.258671] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=19][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=38, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:29.258719] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=32] will sleep(sleep_us=38000, remain_us=1143844, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=38, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:29.272153] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=69] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:29.294940] INFO eloop_run (eloop.c:140) [13674][pnio2][T0][Y0-0000000000000000-0-0] [lt=74] PNIO [ratelimit] time: 1713775589294938, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007557, add_bytes: 0 [2024-04-22 16:46:29.296870] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=22][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:38, local_retry_times:38, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:29.296958] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=66][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.296980] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=21][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.296991] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:29.297050] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:29.300993] WDIAG setup_next_scanner (ob_direct_receive_op.cpp:204) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=37][errcode=-4725] [2024-04-22 16:46:29.301048] WDIAG [SQL.EXE] setup_next_scanner (ob_direct_receive_op.cpp:207) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=52][errcode=-4725] error occurring in the remote sql execution, please use the current TRACE_ID to grep the original error message on the remote_addr.(ret=-4725, remote_addr="172.23.88.129:2882") [2024-04-22 16:46:29.301083] WDIAG [SQL.EXE] inner_open (ob_direct_receive_op.cpp:53) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=32][errcode=-4725] failed to setup first scanner(ret=-4725) [2024-04-22 16:46:29.301103] WDIAG [SQL.ENG] open (ob_operator.cpp:770) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=17][errcode=-4725] Open this operator failed(ret=-4725, op_type="PHY_DIRECT_RECEIVE") [2024-04-22 16:46:29.301117] WDIAG [SQL.EXE] open (ob_execute_result.cpp:104) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=13][errcode=-4725] open operator failed(ret=-4725) [2024-04-22 16:46:29.301129] WDIAG [SQL] open_result (ob_result_set.cpp:211) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] fail open main query(ret=-4725) [2024-04-22 16:46:29.301140] WDIAG [SQL] open (ob_result_set.cpp:160) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=10][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.301190] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=9] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.301231] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=34][errcode=-4725] open result set failed(ret=-4725) [2024-04-22 16:46:29.301257] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:699) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=23][errcode=-4725] result set open failed(ret=-4725, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-04-22 16:46:29.301275] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:834) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=16][errcode=-4725] execute failed(ret=-4725, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=39, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-04-22 16:46:29.301310] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=18] will sleep(sleep_us=39000, remain_us=1101253, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=39, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:29.340466] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=20][errcode=-4725] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:39, local_retry_times:39, err_:-4725, err_:"OB_TABLET_NOT_EXIST", retry_type:1, client_ret:-4725}, need_retry=true) [2024-04-22 16:46:29.340556] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=67][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.340578] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=20][errcode=-4725] result set close failed(ret=-4725) [2024-04-22 16:46:29.340590] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:839) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=11][errcode=-4725] failed to close result(close_ret=-4725, ret=-4725) [2024-04-22 16:46:29.340617] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:869) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=13][errcode=-4725] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4725, ret=-4725) [2024-04-22 16:46:29.341279] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:202) [14049][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=49] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f4d98ddc2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-04-22 16:46:29.344391] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.344424] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=12337504, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=218, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:29.344389] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.344506] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=40000, remain_us=1058057, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=40, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:29.388728] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.388820] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=41000, remain_us=1013743, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=41, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:29.389056] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [13640][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=60] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=62914, map_replace_skip_count_=2) [2024-04-22 16:46:29.430029] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=52][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:29.431767] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:290) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-04-22 16:46:29.431804] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="172.23.88.131:2882", cluster_service_tablet_id={id:226}) [2024-04-22 16:46:29.431851] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:1713775589162392536, v:0}, valid_part_count=1, total_part_count=1, generate_timestamp=1713775589431758) [2024-04-22 16:46:29.431865] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [14056][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1713775589231755, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-04-22 16:46:29.433778] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.433869] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=42000, remain_us=968694, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=42, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:29.433904] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [13720][SysLocAsyncUp0][T0][YB42AC175883-000616AB07685575-0-0] [lt=64][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1713772869, tenant_id:1, ls_id:{id:1}, renew_for_tenant:true, add_timestamp:1713775589433742}) [2024-04-22 16:46:29.444589] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=9][errcode=-4725] REACH SYSLOG RATE LIMIT [bandwidth] [2024-04-22 16:46:29.448354] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.448454] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=12233476, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=219, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:29.462311] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:588) [14020][T1_TntSharedTim][T1][Y0-0000000000000000-0-0] [lt=64] dump dtl interm result cost(us)(dump_cost=4254, ret=0, interm count=0, dump count=191) [2024-04-22 16:46:29.465602] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:603) [14020][T1_TntSharedTim][T1][Y0-0000000000000000-0-0] [lt=68] clear dtl interm result cost(us)(clear_cost=3218, ret=0, gc_.expire_keys_.count()=0, dump count=191, interm count=0, clean count=0) [2024-04-22 16:46:29.472924] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1127) [13639][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=63] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-20811939840, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:4, 500, 508, 509, 1]) [2024-04-22 16:46:29.479771] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.479860] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=43000, remain_us=922702, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=43, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:29.522052] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [14039][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=65] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-04-22 16:46:29.525702] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.525793] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=44000, remain_us=876769, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=44, v.err_=-4725, timeout_timestamp=1713775590402561) [2024-04-22 16:46:29.554025] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=1] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.554116] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [14009][T1_FreInfoReloa][T1][YB42AC175883-000616AB083732A7-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=12127813, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=220, v.err_=-4725, timeout_timestamp=1713775601681928) [2024-04-22 16:46:29.560521] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [13778][TsMgr][T0][Y0-0000000000000000-0-0] [lt=119] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1713775589059038], gts:1713775589060503201, latest_srr:[mts=1713775589560427]}) [2024-04-22 16:46:29.560590] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:179) [13778][TsMgr][T0][Y0-0000000000000000-0-0] [lt=65] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2024-04-22 16:46:29.561408] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [13672][pnio1][T0][YB42AC175883-000616AB095740C9-0-0] [lt=112] ts response task statistics(alloc_count=14374, free_count=7186) [2024-04-22 16:46:29.570127] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8242) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=83][errcode=0] resolve dblink name(has_dblink_node=false, dblink_name=, is_reverse_link=false) [2024-04-22 16:46:29.570248] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=110] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:29.570289] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:8603) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=27] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-04-22 16:46:29.570317] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:8561) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=24][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-04-22 16:46:29.570334] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:8406) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=15][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-04-22 16:46:29.570350] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:8251) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-04-22 16:46:29.570366] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2263) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=11] database not exist(tenant_id=1, database_name=oceanbase) [2024-04-22 16:46:29.570377] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3067) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=9][errcode=-5019] fail to resolve table(ret=-5019) [2024-04-22 16:46:29.570388] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3122) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=10][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-04-22 16:46:29.570404] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3218) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-04-22 16:46:29.570416] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:15163) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-04-22 16:46:29.570426] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:15087) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-04-22 16:46:29.570436] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:3675) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=9][errcode=-5019] resolve basic table failed(ret=-5019) [2024-04-22 16:46:29.570456] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3614) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=18][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-04-22 16:46:29.570468] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1076) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-04-22 16:46:29.570478] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1278) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2024-04-22 16:46:29.570489] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:187) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-04-22 16:46:29.570510] WDIAG [SQL] generate_stmt (ob_sql.cpp:2840) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2024-04-22 16:46:29.570523] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:2961) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:29.570537] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4811) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-04-22 16:46:29.570548] WDIAG [SQL] handle_text_query (ob_sql.cpp:2560) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-04-22 16:46:29.570559] WDIAG [SQL] stmt_query (ob_sql.cpp:227) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-04-22 16:46:29.570583] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:979) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=11][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-04-22 16:46:29.570605] WDIAG [SERVER] do_process (obmp_query.cpp:760) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=20][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-04-22 16:46:29.570655] WDIAG [SERVER] do_process (obmp_query.cpp:884) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=13][errcode=-5019] query failed(ret=-5019, session={this:0x7f4d7edd01f8, id:3221225477, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-04-22 16:46:29.570713] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [14106][T1_L0_G0][T1][YB42AC175883-000616AB0867352B-0-0] [lt=42] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x5d1c83c 0xaee7e40 0xae94a45 0x579c808 0x578fbfb 0x5788b5f 0x577a048 0xabd78a4 0x16c398be 0x7f4dc8ea7ea5 0x7f4dc8bd0b0d") [2024-04-22 16:46:29.573542] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:441) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=0] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=true, tablet_list=[{id:1}], ls_ids=[], error_code=-4725) [2024-04-22 16:46:29.573726] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [13685][SerScheQueue0][T1][YB42AC175883-000616AB09D7330F-0-0] [lt=1] will sleep(sleep_us=45000, remain_us=828836, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=45, v.err_=-4725, timeout_timestamp=1713775590402561)